Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.
2023-July-11 Service Incident 2
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday July 11th 2023, 16:08- 18:43 UTC
What happened:
Approximately 50% of virtual desktop, emulator, and simulator tests in the US-WEST region experienced slowness (30s - 90s) in their execution.
Why it happened:
The data store used for session management reached CPU saturation, causing high response times. This service was handling a portion of all tests. Tests routing through the stable service were not impacted.
How we fixed it:
We configured all tests to route through the stable service.
What we are doing to prevent it from happening again:
We are investigating the best approach for session management caching, as well as further optimizations that can be made. We are capturing additional metrics and adding appropriate alerting.
Resolved
We have taken remedial action and are now seeing virtual desktop, emulator and simulator tests in the US Data Center execute as expected. All services are fully operational.
Investigating
We are seeing virtual desktop, emulator and simulator test slowness in the US Data Center, causing an increase in test duration timeouts. We are investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), and Visual Testing (Visual Testing Infrastructure).