Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2024-March-11 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Saturday March 9th 2024, 20:00 UTC - Monday March 11th, 13:22 UTC
What happened:
Some Windows, Linux, and Android Emulator workloads in our US-West-1 data center experienced elevated error rates.
Why it happened:
A hardware instruction silently failed to be programmed in a network device’s TCAM. This resulted in a subset of traffic from one of our data centers being unable to reach the public Internet.
How we fixed it:
We forced a reprogramming of the TCAM instructions in question.
What we are doing to prevent it from happening again:
We are investigating ways to better understand silent failures in this portion of our infrastructure.
Resolved
After taking remedial action, we are seeing error rates return to normal levels. This incident is now resolved.
Update
We are seeing elevated error rates in Desktop tests in the US-West-1 datacenter. We continue to investigate.
Investigating
We are currently seeing elevated error rates in Desktop tests in the US-West-1 datacenter. We are investigating.
This incident affected: Automated Browser Testing (US-West) and Live Browser Testing (US-West).