Maintenance Windows for Sauce Labs
We are experiencing cellular connectivity issues affecting 10% of private RDC devices in our US datacenter. A carrier technician is currently on site investigating the issue.
2023-April-6 Service Incident 2
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday April 6th 2023, 21:52 - 23:44 UTC
What happened:
The service responsible for starting tunnels and assigning jobs to tunnels failed due to increased memory usage. This affected all jobs in the US West data center as they needed to be checked for tunnel access.
Why it happened:
There was a bottleneck in our tunnel services that caused increased memory usage under heavy load.
How we fixed it:
We modified the tunnel services to distribute requests better, and to reduce memory consumption when those requests were being made.
What we are doing to prevent it from happening again:
The initial condition causing the increased memory usage on the tunnel services has been resolved, and we will improve our monitoring to alert earlier on high memory usage conditions.
Resolved
This incident has been resolved. All services are fully operational.
Monitoring
We have taken remedial action and are beginning to see improvements starting new tunnels and running tests using existing tunnels. We are monitoring.
Update
We are seeing an increased rate of test failure in our US West DataCenter affecting both tunneled and non-tunneled tests. We continue to investigate.
Investigating
We are experiencing issues with starting tunnels and running tests using existing tunnels in our US West Data Center. We are investigating.
This incident affected: Automated Browser Testing (US-West), Automated Virtual Mobile Device Testing (US-West), Automated Real Device Testing (US-West), Live Browser Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), API Testing (US-West), Sauce Connect (US-West), and Native Framework Mobile App Testing (US-West).