Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2022-October-12 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday October 12th 2022, 19:10 - 19:40 UTC
What happened:
Real device tests in our US West region failed due to connectivity issues between RDC services (test orchestration) running in GCP and the data center where our physical devices reside. These connectivity issues lasted approximately 30 minutes.
Why it happened:
It still needs to be determined where the network connectivity issue started. We are still working with Google to research the cause, and most indicators point to an issue on their end.
How we fixed it:
The network connectivity restored itself.
What we are doing to prevent it from happening again:
We are increasing the frequency we run our end-to-end tests in production to detect these incidents faster.
Resolved
We have seen all real device testing stabilized for the last 60 minutes in our US-West-1 data center. All services are fully operational.
Monitoring
We are seeing real device tests in the US-West-1 datacenter working as expected again. We are monitoring.
Investigating
We are seeing real device tests in the US-West-1 datacenter failing. We are investigating.
This incident affected: Automated Real Device Testing (US-West).