Partially Degraded Service
We are currently experiencing a slight reduction in available public and private iOS devices in the EU-Central Data Center. We expect to restore full availability for all iOS users in the EU-Central Data Center by tomorrow, Thursday, July 3, 2025, end of day
2023-October-26 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday October 26th 2023, 17:03 - 17:30 UTC
What happened:
For the duration of the incident, customers could not upload new files or list files/groups through our Application Management UI. Some users received an HTTP error code of 429 due to them being rate-limited.
Why it happened:
Customers began to have issues uploading assets to our storage service due to a high number of concurrent uploads. This spike in upload requests caused a delay in acquiring a backend connection for the storage service.
How we fixed it:
We recycled the storage service pods, which cleared the connectivity issue with our backend service.
What we are doing to prevent it from happening again:
We are looking at the following areas:
- Determining how better to handle periods of increased load for our storage service.
- Refining our alerting for our storage service to detect performance issues customers may experience when interacting with it.
Resolved
After taking remedial action, RDC and Emulator/Simulator Live and Automated tests on our US-West Data Center are back to normal. All services are fully operational.
Investigating
We are aware of an ongoing issue related to Application Management impacting RDC and Emulator/Simulator Live and Automated tests on our US-West Data Center. We're investigating
This incident affected: Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), API Testing (US-West), and Native Framework Mobile App Testing (US-West).