Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2024-March-11 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Saturday March 9th 2024, 20:00 UTC - Monday March 11th, 13:22 UTC
What happened:
Some Windows, Linux, and Android Emulator workloads in our US-West-1 data center experienced elevated error rates.
Why it happened:
A hardware instruction silently failed to be programmed in a network device’s TCAM. This resulted in a subset of traffic from one of our data centers being unable to reach the public Internet.
How we fixed it:
We forced a reprogramming of the TCAM instructions in question.
What we are doing to prevent it from happening again:
We are investigating ways to better understand silent failures in this portion of our infrastructure.
Resolved
After taking remedial action, we are seeing error rates return to normal levels. This incident is now resolved.
Update
We are seeing elevated error rates in Desktop tests in the US-West-1 datacenter. We continue to investigate.
Investigating
We are currently seeing elevated error rates in Desktop tests in the US-West-1 datacenter. We are investigating.
This incident affected: Automated Browser Testing (US-West) and Live Browser Testing (US-West).