Maintenance Windows for Sauce Labs
We are aware of and working to resolve intermittent WiFi connectivity and slowness issues affecting Android Real Devices in our US-West-1 data center. During a maintenance window on September 25, we made changes to our wireless infrastructure which has greatly improved the situation, but we are still monitoring and working on further improvements. Please contact our Support team should you wish to report any such issues.
2021-November-02 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Tuesday November 2nd 2021, 08:50 - 11:14 UTC
What happened:
A certificate expired that caused an error in our health checks for iOS devices. All iOS devices in EU and US were put into an unhealthy state and were not available for testing.
Why it happened:
The renewed certificate was not deployed in time.
How we fixed it:
We deployed a change with the updated certificate.
What we are doing to prevent it from happening again:
We are changing our policies in order to ensure that the update for these certificates happens at least two weeks prior to their expiration date. We have also made changes to ensure that if one of these certificates does expire again in the future, only a small subset of our iOS devices will be affected.
Resolved
This incident has been resolved. Device availability has returned to normal levels.
Monitoring
We have identified the issue and taken corrective action. Device availability is gradually returning to normal levels.
Investigating
We are seeing a high number of unavailable iOS real devices in our US-West and EU-Central data centers. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central) and Live Real Device Testing (US-West, EU-Central).