Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.
2025-January-21 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday January 21st 2025, 12:40 - 13:44 UTC
What happened:
Tests using iOS Real Devices experienced failures to download and install apps for the eu-central01 and us-west-1 datacenters.
Why it happened:
A misconfiguration with the resigner service caused errors when communicating with the app storage.
How we fixed it:
A rollback was executed, restoring configuration to the previous working state.
What we are doing to prevent it from happening again:
- Improve canary deployments for each Real Device Cloud region.
- Improve end to end testing for canary deployments.
- Improve SLOs for the Real Device application installs
Resolved
After taking remedial action, apps are now installing correctly in all datacenters. This incident is resolved.
Investigating
We are currently seeing App installation errors when trying to run iOS tests on our US-West-1 & EU-Central-1 Datacenter. We are investigating
This incident affected: Automated Virtual Mobile Device Testing (US-West, EU-Central), Automated Real Device Testing (US-West, EU-Central), Live Virtual Mobile Device Testing (US-West, EU-Central), Live Real Device Testing (US-West, EU-Central), and Native Framework Mobile App Testing (US-West, EU-Central).