Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-November-28 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Monday November 27 2023, 20:00 - Tuesday November 28 2023, 15:36 UTC
What happened:
Pixel devices in Europe failed to recover from a specific test scenario, resulting in decreased availability of Pixel devices over time.
Why it happened:
A combination of defects resulted in devices failing to be properly cleaned following a specific customer test scenario involving audio capture.
How we fixed it:
Multiple software defects were fixed and deployed.
What we are doing to prevent it from happening again:
The defects were resolved. Additionally, new observability work is under investigation to improve detection time for similar availability situations in the future.
Resolved
After taking remedial action, all Google Pixel devices are now available in the EU-Central-1 data center.
Identified
Public Google Pixel devices continue to be unavailable in the EU-Central-1 data center. We have identified the root cause and are taking remedial action.
Investigating
Public Google Pixel devices are currently unavailable in the EU-Central-1 data center. We are investigating.
This incident affected: Automated Real Device Testing (EU-Central) and Live Real Device Testing (EU-Central).