Tuesday May 16th 2023, 04:10 - 06:00 UTC
Internal services experienced high response times in short intervals due to an authentication service being hit with a burst of high traffic. This caused Sauce Connect tunnels to shut down, which triggered customer scripts to restart their tunnels automatically.
Increased traffic from an internal service to an authentication service caused strain on the data store used by the authentication service.
The following steps we taken to restore service:
We are working with the datastore vendor to make the service more resilient and performant under the conditions experienced during the incident.