Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-June-10 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday, June 10th, 06:13 - 08:58 UTC
What happened:
Android real devices had elevated error rates and some unavailability in our US-West-1 and EU-Central-1 data centers, causing some tests to fail.
Why it happened:
Memory spikes were observed in our Android device pools. We also found that two sets of a device pool pod were running, causing some internal endpoint failures.
How we fixed it:
The remediation here was to reboot the hardware hosting the affected service, which resolved the issue with the multiple pods running. To address the memory spikes, we rolled back to a previously deployed RDC service version.
What we are doing to prevent it from happening again:
We are analyzing the code changes that could have caused this to learn how we can avoid this issue in the future.
Resolved
All eu-central-1 and us-west-1 RDC Android tests should be operating normally.
Update
We have taken remedial action and are monitoring the state of RDC Android testing in us-west-1 and eu-central-1.
Monitoring
RDC Android devices are experiencing elevated error rates and unavailability in us-west-1 and eu-central-1. We are taking remedial action. RDC Android tests may fail during this time.
This incident affected: Automated Real Device Testing (US-West, EU-Central).