Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-July-11 Service Incident 2
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday July 11th 2023, 16:08- 18:43 UTC
What happened:
Approximately 50% of virtual desktop, emulator, and simulator tests in the US-WEST region experienced slowness (30s - 90s) in their execution.
Why it happened:
The data store used for session management reached CPU saturation, causing high response times. This service was handling a portion of all tests. Tests routing through the stable service were not impacted.
How we fixed it:
We configured all tests to route through the stable service.
What we are doing to prevent it from happening again:
We are investigating the best approach for session management caching, as well as further optimizations that can be made. We are capturing additional metrics and adding appropriate alerting.
Resolved
We have taken remedial action and are now seeing virtual desktop, emulator and simulator tests in the US Data Center execute as expected. All services are fully operational.
Investigating
We are seeing virtual desktop, emulator and simulator test slowness in the US Data Center, causing an increase in test duration timeouts. We are investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), and Visual Testing (Visual Testing Infrastructure).