Sauce Labs Maintenance Windows for Sauce Labs
We are seeing issues running automated tests in our Apac-Southeast-1 Data Center, we are currently investigating.

2022-May-26 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Thursday May 26th 2022, 17:37 - 18:18 UTC

What happened:

In our US-West and US-East data centers, the operation of automated Virtual Device and Real Device tests was impacted after a third party migrated part of our cloud infrastructure to a new platform service.

Why it happened:

After investigation, Sauce Labs identified that the third party had rate limited our traffic to 100Mbps on our interconnects after the migration. This caused communication between our Cloud infrastructure provider and on-premise nodes to intermittently fail due to bandwidth contention problems. Normally traffic over a single interconnect is ~1Gbps utilization. 

How we fixed it:

The incident was resolved when the third party reverted their migration changes of our components inside our Cloud infrastructure.

What we are doing to prevent it from happening again:

We asked the third party to migrate some interconnects for us to a new platform so that we could leverage BFD, which would give us faster failure detection times, which would improve the reliability of hybrid communications.

Posted Jun 09, 2022 - 15:27 UTC

Resolved
After taking remedial measures we are seeing all testing resuming back to normal levels on all data centers. All services are fully operational.
Posted May 26, 2022 - 18:42 UTC
Investigating
We are seeing all Virtual Desktop and Mobile tests having errors starting up on our us-west-1 and us-east-1. We are investigating.
Posted May 26, 2022 - 18:13 UTC
This incident affected: Automated Browser Testing (US-West, US-East (Headless)), Automated Virtual Mobile Device Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), Automated Real Device Testing (US-West), Native Mobile App Testing (US-West), Live Browser Testing (US-West), and Visual Testing Infrastructure.