Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2024-09-15 Resolved Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Saturday September 15th, 2024 17:02 - 20:15 UTC
What happened:
Our Real Device (RDC) availability dropped below 80% across our US West and US East regions.
Why it happened:
The issue was caused by a failure in our certificate renewal process set to occur automatically, but failed to run as expected.
How we fixed it:
We fixed the problem by restarting the impacted service, which automatically renewed the necessary certificates. This brought our device availability back to normal.
What we are doing to prevent it from happening again:
We have initiated an investigation to review and fix the renewal process script to ensure it functions correctly in the future.
Resolved
Between 17:02 and 20:15 UTC, automated tests executed on our real devices in the US West Data Center experienced an increased error rate due to reduced availability of the devices. The issue has been identified. All services are back to normal.