Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-Feb-19 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Friday, February 12, 2021, 11:30 PST - Monday, February 15, 2021 14:10 PST (first occurrence)
Thursday, February 18, 2021, 21:00 PST - Friday, February 19, 2021 13:30 PST (second occurrence)
What happened:
IPSec VPN tunnels in the US-West region timed out.
Why it happened:
A monitoring change that was introduced led to high CPU usage and resource exhaustion on our IPSec concentrator cluster in the US-West-1 data center.
How we fixed it:
We restarted a process on the IPSec concentrator cluster in the US-West-1 data center and also rolled back the monitoring change that triggered the event.
What are we doing to prevent it from happening again:
We are re-evaluating our monitoring implementation for this product and working with the vendor to address software issues.
Resolved
IPSec/VPN Tunnels are now functional, the issue has been identified and resolved.
Investigating
IPSec/VPN Tunnels in our US-West-1 data center are currently experiencing partial disruption in service. We are taking remedial action.
This incident affected: IPSec VPN (US-West).