Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2024-January-19 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday January 19th 2024, 20:00 - 21:59 UTC
What happened:
An issue with route programming within a service mesh resulted in traffic being dropped to new pods involved in the Sauce Orchestrate request flow.
Why it happened:
A service mesh bug resulted in some routes not being programmed.
How we fixed it:
The control plane of the service mesh was restarted to force a sync event.
What we are doing to prevent it from happening again:
We are improving monitoring for the service mesh and the affected services. Additionally, we are developing a plan for upgrades of the service mesh across the fleet.
Resolved
After taking remedial action, we are seeing normal execution of Sauce Orchestrate tests on the US-West-1 data center. This incident is resolved.
Update
We are currently seeing elevated error rates when running tests via Sauce Orchestrate in our US-West-1 datacenter, we are still investigating.
Investigating
We are currently seeing elevated error rates when running tests via Sauce Orchestrate in our US-West-1 datacenter, we are currently investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), and Automated Real Device Testing (US-West).