Maintenance Windows for Sauce Labs
We are aware of and working to resolve intermittent WiFi connectivity and slowness issues affecting Android Real Devices in our US-West-1 data center. During a maintenance window on September 25, we made changes to our wireless infrastructure which has greatly improved the situation, but we are still monitoring and working on further improvements. Please contact our Support team should you wish to report any such issues.
2021-September-19 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Sunday September 19th 2021, 15:42 - 16:35 UTC
What happened:
Automated and Live Mac and iOS tests in our EU-Central-1 data center were failing to start.
Why it happened:
We experienced a combination of router software failure caused by a third-party bug as well as with a failure to properly route traffic.
How we fixed it:
The issue was resolved by initiating a manual failover of traffic to the secondary circuit.
What we are doing to prevent it from happening again:
We are investigating potential options to avoid the failure to automatically failover traffic routing in the future. Additionally, we are removing a configuration from our edge routers to ensure we do not hit this particular software bug again.
Resolved
We have taken remedial measures and all Mac testing is back and running. All services are fully operational.
Investigating
Mac tests (Live and Automated) are not starting in our EU data center. We are investigating.
This incident affected: Live Browser Testing (US-West) and Automated Browser Testing (US-West).