Sauce Labs Maintenance Windows for Sauce Labs
We are experiencing an intermittent issue in our US-West-1 datacenter that causes elevated wait times and error rates when starting desktop browser and virtual mobile device tests. Until we address the underlying causes of this issue, we expect this issue to continue to reoccur. We will post incidents here on our status page as each instance of this issue occurs whilst we continue to work to address the underlying cause.
2023-Feb-27 Service Incident
Incident Report for Sauce Labs
Postmortem

Dates:

Friday February 27th 2023, 03:00 - 14:32 UTC

What happened:

A failure on the primary node in our European based IPsec cluster caused customer tests (using IPSec tunnels) to fail in that region. Throughout the incident, tests that had a destination within a customer network were intermittently blackholed.

Why it happened:

Our IPsec Tunnel health checks failed because BGP was down towards the primary IPSec firewall cluster node, which caused network errors on the IPsec cluster when tests were attempting to reach customer networks. 

How we fixed it:

The secondary node in the IPSec cluster was first rebooted to ensure that this node did not have any failures. After verifying services would remain stable, we initiated a failover from the primary to the secondary node, which resulted in restoring BGP on the cluster.

What we are doing to prevent it from happening again:

A change scheduled for the March 25 EU maintenance window will involve upgrading the IPSec firewall in our Europe region, as suggested by the vendor. We also performed the same maintenance for our US region’s IPSec firewall during the US maintenance window on March 18.

Posted Mar 21, 2023 - 14:26 UTC

Resolved
After taking remedial action, connectivity has been restored to IPSec tunnels in the EU-Central-1 datacenter. This issue is now resolved.
Posted Feb 27, 2023 - 15:20 UTC
Monitoring
After taking remedial action, connectivity has been restored to IPSec tunnels in the EU-Central-1 datacenter. We are currently monitoring.
Posted Feb 27, 2023 - 14:45 UTC
Investigating
We are experiencing issues with tests using IPSec Tunnels in the EU-Central-1 datacenter. we are currently investigating.
Posted Feb 27, 2023 - 13:24 UTC
This incident affected: IPSec VPN (EU-Central).