Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-November-02 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday November 2nd 2021, 08:50 - 11:14 UTC
What happened:
A certificate expired that caused an error in our health checks for iOS devices. All iOS devices in EU and US were put into an unhealthy state and were not available for testing.
Why it happened:
The renewed certificate was not deployed in time.
How we fixed it:
We deployed a change with the updated certificate.
What we are doing to prevent it from happening again:
We are changing our policies in order to ensure that the update for these certificates happens at least two weeks prior to their expiration date. We have also made changes to ensure that if one of these certificates does expire again in the future, only a small subset of our iOS devices will be affected.
Resolved
This incident has been resolved. Device availability has returned to normal levels.
Monitoring
We have identified the issue and taken corrective action. Device availability is gradually returning to normal levels.
Investigating
We are seeing a high number of unavailable iOS real devices in our US-West and EU-Central data centers. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central) and Live Real Device Testing (US-West, EU-Central).