Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2023-September-7 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Wednesday September 6th 2023, 18:35 UTC - Thursday September 7th 19:05 UTC

What happened:

A subset of customers were unable to run tests with specific uploaded apps in the EU region intermittently for the duration of the incident. 

Why it happened:

During migrating users from regional to global teams, several organizations in the EU had their users moved to the default team, meaning they were unable to run tests using apps uploaded by another team. Various scripts were created to prepare and execute this migration, one of which was meant to determine a user’s region. This script had a bug in it and misconfigured 55 organizations.

How we fixed it:

We were able to restore the user configuration to restore service, and then we addressed the bug in the script and completed the migration. 

What we are doing to prevent it from happening again:

We have addressed bugs in the migration pipeline and the scripts it leverages for future runs.

Posted Sep 13, 2023 - 10:30 UTC

Resolved

After taking remedial action, impacted subset of user accounts are restored to correct teams. All services are fully operational.
Posted Sep 07, 2023 - 19:06 UTC

Identified

Sauce Labs accounts assigned to custom teams were moved to the default team for a small subset of customers. We have identified the root cause and are taking remedial action.
Posted Sep 07, 2023 - 15:03 UTC
← Current Status Powered by Atlassian Statuspage