Sauce Labs Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2023-September-7 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Wednesday September 6th 2023, 18:35 UTC - Thursday September 7th 19:05 UTC

What happened:

A subset of customers were unable to run tests with specific uploaded apps in the EU region intermittently for the duration of the incident. 

Why it happened:

During migrating users from regional to global teams, several organizations in the EU had their users moved to the default team, meaning they were unable to run tests using apps uploaded by another team. Various scripts were created to prepare and execute this migration, one of which was meant to determine a user’s region. This script had a bug in it and misconfigured 55 organizations.

How we fixed it:

We were able to restore the user configuration to restore service, and then we addressed the bug in the script and completed the migration. 

What we are doing to prevent it from happening again:

We have addressed bugs in the migration pipeline and the scripts it leverages for future runs.

Posted Sep 13, 2023 - 10:30 UTC

Resolved
After taking remedial action, impacted subset of user accounts are restored to correct teams. All services are fully operational.
Posted Sep 07, 2023 - 19:06 UTC
Identified
Sauce Labs accounts assigned to custom teams were moved to the default team for a small subset of customers. We have identified the root cause and are taking remedial action.
Posted Sep 07, 2023 - 15:03 UTC