Maintenance Windows for Sauce Labs
Sauce Labs has consolidated onto one status page for all data centers.
The old status pages for US-West, US-East and EU-Central are still available to be able to view incident history. Anyone who was subscribed to notifications from one of the old status pages has been migrated onto this consolidated page. If you received an email saying that you had been unsubscribed from notifications, that was from one of the old pages and can be disregarded.
If you would like to select which components that you receive notifications for (e.g. if you are only interested in receiving notifications about incidents affecting Real Device Testing), you can access the subscription management page by clicking "Subscribe To Updates" above and enter the email address you are subscribed with. Additionally there is a "Manage Your Subscription" link on any email you will receive from the status page going forward.
The old status pages for US-West, US-East and EU-Central are still available to be able to view incident history. Anyone who was subscribed to notifications from one of the old status pages has been migrated onto this consolidated page. If you received an email saying that you had been unsubscribed from notifications, that was from one of the old pages and can be disregarded.
If you would like to select which components that you receive notifications for (e.g. if you are only interested in receiving notifications about incidents affecting Real Device Testing), you can access the subscription management page by clicking "Subscribe To Updates" above and enter the email address you are subscribed with. Additionally there is a "Manage Your Subscription" link on any email you will receive from the status page going forward.
2021-March-20 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates
Monday, March 22, 6:24am UTC - Monday, March 22nd, 2:32pm UTC
What happened:
Sauce Visual tests during this timeframe queued but did not run, making builds timeout and fail.
Why it happened:
The service in charge of managing jobs queues was hanging in a loop state after reaching an edge case introduced by a new feature.
How we fixed it:
The issue was remediated and an update was deployed.
What we are doing to prevent it from happening again:
We’re increasing system visibility and robustness by adding new alerts and health checks for the failing service.
Resolved
This incident has been resolved.
Monitoring
A fix has been implemented. Previously queued tests will have failed but new Visual tests are running correctly We are continuing to monitor the situation.
Investigating
Visual tests are queued up and are currently not running. We are investigating.
This incident affected: Visual Testing Hub, Visual Testing Infrastructure, and Visual Testing REST API.