Maintenance Windows for Sauce Labs
Sauce Labs will have a four hour previously scheduled maintenance window on Wednesday, July 10th, starting at 04:00 UTC and ending at 08:00 UTC. During this maintenance window, we will be making updates to our infrastructure and services in our US West data center. These maintenance actions may cause portions of the service (including running automated and manual tests) to be unavailable for up to four hours.
2023-October-4 Resolved Service Incident (2)
Incident
Report for Sauce Labs
Postmortem
Dates:
Wednesday October 4th 2023, 11:00 - 12:01 UTC
What happened:
Customers may have experienced session creation errors when testing with Android Emulators in our EU-Central and US-West data centers.
Why it happened:
The issue was caused by deploying a component responsible for starting sessions in our virtual infrastructure. The code of the new component version contained an error that caused the session processing code to return an HTTP 500 (internal server error) message.
How we fixed it:
When we identified the change causing the issue, we immediately reverted to the previous stable component version. After that, we deployed that component with the latest version, containing a fix.
What we are doing to prevent it from happening again:
- We are adding better static code analysis to the given component, which should prevent errors similar to the one causing this incident.
- We are improving test code coverage of that part of the component.
- We are fine-tuning our alerts so they will alert us much quicker for similar scenarios (too many HTTP 500 returned).
Resolved
Between approximately 11:00 and 12:00 UTC, customers may have experienced session creation errors when testing with Android Emulators in our EU-Central and US-West data centers. This issue has now been resolved.