2022-November-7 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Friday November 7th 2022, 08:30 - 19:30 UTC

What happened:

A portion of newly signed-up users saw an error in the Sauce Labs frontend as they tried to log in after signing up.

Why it happened:

As new users were signing up in Sauce Labs, a backend service responsible for managing user data wasn’t being updated fast enough. This caused the user’s subsequent login attempts to fail until that data was updated.

This race condition was introduced as a change to the service managing user concurrency was made and pushed to production.

How we fixed it:

That change was reverted to address the race condition, and the team also changed to the front end to respond more gracefully in case a backend service error is thrown, as we saw in this incident.

What we are doing to prevent it from happening again:

The team is exploring more areas where the Sauce Labs front end could run into this scenario and addressing those areas.

Posted Dec 22, 2022 - 15:03 UTC

Resolved

We have taken remedial action and signing up for a new account is back to normal. All services are fully operational.

Posted Nov 07, 2022 - 20:40 UTC

Monitoring

We are now seeing 500 errors residing for new accounts. We are monitoring.

Posted Nov 07, 2022 - 19:44 UTC

Update

Users are seeing 500 errors when signing up for a new account in our US and EU datacenters. We are continuing to investigate this issue.

Posted Nov 07, 2022 - 18:42 UTC

Investigating

Users are seeing 500 errors when signing up for a new account in our US and EU datacenters.
We are investigating.

Posted Nov 07, 2022 - 18:34 UTC

This incident affected: Sauce Labs Dashboard (US-West, EU-Central) and Other (saucelabs.com).