Maintenance Windows for Sauce Labs
Sauce Labs has consolidated onto one status page for all data centers.
The old status pages for US-West, US-East and EU-Central are still available to be able to view incident history. Anyone who was subscribed to notifications from one of the old status pages has been migrated onto this consolidated page. If you received an email saying that you had been unsubscribed from notifications, that was from one of the old pages and can be disregarded.
If you would like to select which components that you receive notifications for (e.g. if you are only interested in receiving notifications about incidents affecting Real Device Testing), you can access the subscription management page by clicking "Subscribe To Updates" above and enter the email address you are subscribed with. Additionally there is a "Manage Your Subscription" link on any email you will receive from the status page going forward.
The old status pages for US-West, US-East and EU-Central are still available to be able to view incident history. Anyone who was subscribed to notifications from one of the old status pages has been migrated onto this consolidated page. If you received an email saying that you had been unsubscribed from notifications, that was from one of the old pages and can be disregarded.
If you would like to select which components that you receive notifications for (e.g. if you are only interested in receiving notifications about incidents affecting Real Device Testing), you can access the subscription management page by clicking "Subscribe To Updates" above and enter the email address you are subscribed with. Additionally there is a "Manage Your Subscription" link on any email you will receive from the status page going forward.
2021-June-10 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday, June 10th, 06:13 - 08:58 UTC
What happened:
Android real devices had elevated error rates and some unavailability in our US-West-1 and EU-Central-1 data centers, causing some tests to fail.
Why it happened:
Memory spikes were observed in our Android device pools. We also found that two sets of a device pool pod were running, causing some internal endpoint failures.
How we fixed it:
The remediation here was to reboot the hardware hosting the affected service, which resolved the issue with the multiple pods running. To address the memory spikes, we rolled back to a previously deployed RDC service version.
What we are doing to prevent it from happening again:
We are analyzing the code changes that could have caused this to learn how we can avoid this issue in the future.
Resolved
All eu-central-1 and us-west-1 RDC Android tests should be operating normally.
Update
We have taken remedial action and are monitoring the state of RDC Android testing in us-west-1 and eu-central-1.
Monitoring
RDC Android devices are experiencing elevated error rates and unavailability in us-west-1 and eu-central-1. We are taking remedial action. RDC Android tests may fail during this time.
This incident affected: Automated Real Device Testing (US-West, EU-Central).