Sauce Labs Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day

2022-September-8 Resolved Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday September 8th 2022, 17:53 - 19:44 UTC

What happened:

Mobile app uploads were intermittently failing for the duration of the incident. 

Why it happened:

Multiple long-running queries in "list_files" endpoint on our app storage service caused timeouts that made uploading requests fail. Some customers reported approximately 30 minutes of failed attempts to upload mobile apps.

Several database tables had growth that led to their respective indexes becoming stale, causing queries to those tables to perform poorly. This growth was directly correlated to a spike in requests to the service that handles app storage. 

How we fixed it:

We ran a data archival script on the tables in question, rebuilt indexes, and added additional indexes to restore query performance. 

What we are doing to prevent it from happening again:

The team has identified the introduction of rate limiting on this service as a way to control performance in a more orderly fashion.

Posted Oct 15, 2022 - 12:43 UTC

Resolved

Between 17:52 and 19:23 UTC, in our US-West-1 datacenter, there was a problem uploading apps to the Sauce Storage service. During this time the connection would timeout or end with an error. After taking remedial action the service was restored.
Posted Sep 08, 2022 - 04:30 UTC
← Current Status Powered by Atlassian Statuspage