Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2024-February-26 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday, February 26th, 2024 23:08 UTC - Tuesday, February 27th, 2024 02:56 UTC
What happened:
Increased network load which led to test slowness or failures affecting Real Device tests in our US-West-1 data center.
Why it happened:
During the troubleshooting of an increase in network load in our US-West-1 data center, a network failover event was executed..
How we fixed it:
The network failover and additional traffic shaping stabilized services.
What we are doing to prevent it from happening again:
Improvements to the runbook for manually initiated failovers.
Resolved
After taking remedial action, Real device availability and error rates are back to normal in the US-West-1 data center. All services are fully operational.
Monitoring
After taking remedial action, Real device availability and error rates are returning to normal levels in the US-West-1 data center. We are currently monitoring.
Investigating
We are currently seeing elevated error rates and device availability in Real device tests in our US-West-1 datacenter. We are currently investigating
This incident affected: Automated Real Device Testing (US-West), Live Real Device Testing (US-West), and Native Framework Mobile App Testing (US-West).