Maintenance Windows for Sauce Labs
We are investigating an issue with a third-party provider that is causing intermittent "Connection timed out", "pool communication" and "Unknown error while proxying appium request" errors when running tests in our EU and US datacenters. We will be performing emergency maintenance to our EU-Central and US-West datacenters this week to address this issue. Emergency maintenance windows will be posted to this page.
2023-October-26 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday October 26th 2023, 17:03 - 17:30 UTC
What happened:
For the duration of the incident, customers could not upload new files or list files/groups through our Application Management UI. Some users received an HTTP error code of 429 due to them being rate-limited.
Why it happened:
Customers began to have issues uploading assets to our storage service due to a high number of concurrent uploads. This spike in upload requests caused a delay in acquiring a backend connection for the storage service.
How we fixed it:
We recycled the storage service pods, which cleared the connectivity issue with our backend service.
What we are doing to prevent it from happening again:
We are looking at the following areas:
- Determining how better to handle periods of increased load for our storage service.
- Refining our alerting for our storage service to detect performance issues customers may experience when interacting with it.
Resolved
After taking remedial action, RDC and Emulator/Simulator Live and Automated tests on our US-West Data Center are back to normal. All services are fully operational.
Investigating
We are aware of an ongoing issue related to Application Management impacting RDC and Emulator/Simulator Live and Automated tests on our US-West Data Center. We're investigating
This incident affected: Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), API Testing (US-West), and Native Framework Mobile App Testing (US-West).