Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-October-28 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday October 28th 2022, 19:13 - October 29th 2022 00:37 UTC
Friday November 11th 2022, 12:20 - 15:11 UTC
What happened:
Customers using legacy endpoints to fetch concurrency limits and usage observed an elevated number of 504 errors.
Affected endpoints:
- GET .saucelabs.com/rest/v1/users/:user/concurrency
- GET .saucelabs.com/rest/v1.1/users/:user/concurrency
- GET .saucelabs.com/rest/v1.2/users/:user/concurrency
Why it happened:
This increased error rate was caused by connection issues between our ingress routing service and our API gateway service due to an incorrectly configured timeout setting. This timeout setting was introduced during a migration to the new API gateway.
How we fixed it:
The team migrated back to the old API gateway to restore service, and we saw the error rate subside.
What we are doing to prevent it from happening again:
The team reviewed all settings to ensure everything would work as expected as they resumed and finished the migration.
Resolved
After taking remedial action we see all REST services functioning as expected. All services are fully operational.
Update
We continue to investigate the issue affecting REST services across all our data centers including us-east-1. Our dashboard, saucectl, Sauce Connect and REST API methods are affected.
Investigating
We are seeing intermittent 504 errors on several of our REST services in our EU, US and APAC data centers. This affects our UI, Sauce Connect and other REST-based products. We are investigating.
This incident affected: Live Browser Testing (US-West, EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), Live Real Device Testing (US-West, EU-Central), Sauce Connect (US-West, EU-Central), Sauce Labs REST API (US-West, EU-Central), Sauce Labs Dashboard (US-West, EU-Central), Insights (US-West, EU-Central), and Native Framework Mobile App Testing (US-West, EU-Central).