2022-May-26 Service Incident

Incident Report for Sauce Labs

Postmortem

Dates:

Thursday May 26th 2022, 17:37 - 18:18 UTC

What happened:

In our US-West and US-East data centers, the operation of automated Virtual Device and Real Device tests was impacted after a third party migrated part of our cloud infrastructure to a new platform service.

Why it happened:

After investigation, Sauce Labs identified that the third party had rate limited our traffic to 100Mbps on our interconnects after the migration. This caused communication between our Cloud infrastructure provider and on-premise nodes to intermittently fail due to bandwidth contention problems. Normally traffic over a single interconnect is ~1Gbps utilization.

How we fixed it:

The incident was resolved when the third party reverted their migration changes of our components inside our Cloud infrastructure.

What we are doing to prevent it from happening again:

We asked the third party to migrate some interconnects for us to a new platform so that we could leverage BFD, which would give us faster failure detection times, which would improve the reliability of hybrid communications.

Posted Jun 09, 2022 - 15:27 UTC

Resolved

After taking remedial measures we are seeing all testing resuming back to normal levels on all data centers. All services are fully operational.

Posted May 26, 2022 - 18:42 UTC

Investigating

We are seeing all Virtual Desktop and Mobile tests having errors starting up on our us-west-1 and us-east-1. We are investigating.

Posted May 26, 2022 - 18:13 UTC

This incident affected: Automated Browser Testing (US-West, US-East (Headless)), Automated Virtual Mobile Device Testing (US-West), Live Virtual Mobile Device Testing (US-West), Live Real Device Testing (US-West), Automated Real Device Testing (US-West), Native Mobile App Testing (US-West), Live Browser Testing (US-West), and Visual Testing Infrastructure.