Sauce Labs Maintenance Windows for Sauce Labs
We are currently performing maintenance that will affect the initiation of new Sauce Connect tunnel instances. The maintenance is expected to be completed by 2:00 PM PST. We appreciate your understanding and patience as we work to enhance our service.

2023-Feb-27 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Friday February 27th 2023, 03:00 - 14:32 UTC

What happened:

A failure on the primary node in our European based IPsec cluster caused customer tests (using IPSec tunnels) to fail in that region. Throughout the incident, tests that had a destination within a customer network were intermittently blackholed.

Why it happened:

Our IPsec Tunnel health checks failed because BGP was down towards the primary IPSec firewall cluster node, which caused network errors on the IPsec cluster when tests were attempting to reach customer networks. 

How we fixed it:

The secondary node in the IPSec cluster was first rebooted to ensure that this node did not have any failures. After verifying services would remain stable, we initiated a failover from the primary to the secondary node, which resulted in restoring BGP on the cluster.

What we are doing to prevent it from happening again:

A change scheduled for the March 25 EU maintenance window will involve upgrading the IPSec firewall in our Europe region, as suggested by the vendor. We also performed the same maintenance for our US region’s IPSec firewall during the US maintenance window on March 18.

Posted Mar 21, 2023 - 14:26 UTC

Resolved

After taking remedial action, connectivity has been restored to IPSec tunnels in the EU-Central-1 datacenter. This issue is now resolved.
Posted Feb 27, 2023 - 15:20 UTC

Monitoring

After taking remedial action, connectivity has been restored to IPSec tunnels in the EU-Central-1 datacenter. We are currently monitoring.
Posted Feb 27, 2023 - 14:45 UTC

Investigating

We are experiencing issues with tests using IPSec Tunnels in the EU-Central-1 datacenter. we are currently investigating.
Posted Feb 27, 2023 - 13:24 UTC
This incident affected: IPSec VPN (EU-Central).