Sauce Labs Maintenance Windows for Sauce Labs
The APAC-Southeast-1 REST API and app.apac-southeast-1.saucelabs.com UI are returning errors and not usable at the moment. We are taking remedial action
2022-October-25 Resolved Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Tuesday October 25th 2022, 12:32 - 13:33 UTC

What happened:

A small percentage of authentication requests failed causing some users to be unable to log into the Web UI in our EU-Central-1 region. We also observed that some API requests failed during the incident as well. 

Why it happened:

The service that handles authentication requests had one node that was returning errors while attempting to connect to Directory Services. The affected node was using incorrect credentials, which appears to have resulted from a recent deployment of this service. 

How we fixed it:

To re-establish the connection between the Authentication Manager and Directory Services, the team restarted the Authentication Manager nodes. Once this completed, the connection between the services was restored. 

What we are doing to prevent it from happening again:

We have created a new alert that informs teams when this specific condition arises, which will be running continuously but also checked in the post-deployment checks of this service. We are also looking further into how to prevent this password issue in future deployments.

Posted Nov 16, 2022 - 18:18 UTC

Resolved
Between 12:30 UTC and 13:30 UTC some users were unable to login to the Sauce Labs UI. This issue is now resolved, all services are fully operational.
Posted Oct 25, 2022 - 12:30 UTC