Sauce Labs Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2022-November-7 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Friday November 7th 2022, 08:30 - 19:30 UTC

What happened:

A portion of newly signed-up users saw an error in the Sauce Labs frontend as they tried to log in after signing up. 

Why it happened:

As new users were signing up in Sauce Labs, a backend service responsible for managing user data wasn’t being updated fast enough. This caused the user’s subsequent login attempts to fail until that data was updated. 

This race condition was introduced as a change to the service managing user concurrency was made and pushed to production. 

How we fixed it:

That change was reverted to address the race condition, and the team also changed to the front end to respond more gracefully in case a backend service error is thrown, as we saw in this incident. 

What we are doing to prevent it from happening again:

The team is exploring more areas where the Sauce Labs front end could run into this scenario and addressing those areas.

Posted Dec 22, 2022 - 15:03 UTC
Resolved
We have taken remedial action and signing up for a new account is back to normal. All services are fully operational.
Posted Nov 07, 2022 - 20:40 UTC
Monitoring
We are now seeing 500 errors residing for new accounts. We are monitoring.
Posted Nov 07, 2022 - 19:44 UTC
Update
Users are seeing 500 errors when signing up for a new account in our US and EU datacenters. We are continuing to investigate this issue.
Posted Nov 07, 2022 - 18:42 UTC
Investigating
Users are seeing 500 errors when signing up for a new account in our US and EU datacenters.
We are investigating.
Posted Nov 07, 2022 - 18:34 UTC
This incident affected: Sauce Labs Dashboard (US-West, EU-Central) and Other (saucelabs.com).
← Current Status Powered by Atlassian Statuspage