Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-March-18 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday, March 18, 2021 16:48 UTC - 18:08 UTC
What happened:
RDC automated tests failed to upload assets to our EU and US data centers, causing the test reports on the Sauce Labs UI to be displayed as in-progress indefinitely.
Why it happened:
During the course of normal security related work involving policy and group updates, certain permissions related to Amazon S3 access were changed for one of our service accounts, which was incorrectly placed in one of the groups. This prevented the service from uploading assets to AWS.
How we fixed it:
The service account was added to a group that allows read/write access to S3.
What are we doing to prevent it from happening again:
We will create new service accounts in the proper groups and implement the necessary policy changes to ensure the correct access rights are in place.
Resolved
After taking remedial action, all assets are uploaded successfully. All services fully operational.
Investigating
RDC automated tests fail to upload assets on our EU and US datacenter. We are investigating.
This incident affected: Automated Real Device Testing (US-West, EU-Central).