Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-July-22 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday July 22, 7:50 AM - 9:00 AM UTC
What happened:
iOS devices were not available in the US-West Datacenter
Why it happened:
We introduced a new health check, adding more load to the process of re-signing applications. When we deployed a new release the added load brought down the machines hosting the iOS devices.
How we fixed it:
The host machines were restarted and the re-signer services were re-deployed.
What we are doing to prevent it from happening again:
We are improving the application re-signing process to reduce the load on the host machines and added more monitoring to spot extra loads on the re-signers services earlier on.
Resolved
iOS Real Device availability is restored to normal operational levels. This incident is resolved.
Identified
The issue has been identified and a fix implemented. iOS device availability is recovering.
Update
We are continuing to investigate this issue.
Investigating
iOS Real Devices in our US-West Datacenter are currently unavailable. We are taking remedial action.
This incident affected: Automated Real Device Testing (US-West) and Live Real Device Testing (US-West).