Maintenance Windows for Sauce Labs
Customers may experience intermittent errors during automated browser and virtual mobile device tests in our US-West-1 datacenter. We are closely monitoring and investigating the affected services.
2021-September-30 Service Incident
Incident
Report for Sauce Labs
Postmortem
Dates:
Thursday September 30th 2021, 18:00 - Friday October 1st 2021, 01:00 UTC
What happened:
In Visual Testing (Screener), component snapshots were being captured with the “error: certificate has expired” showing on screen. All component tests that were run using ngrok tunnels were affected.
Why it happened:
Screener has a cache layer to avoid downloading resources on every component request. This cache layer has some issues related to an expired third-party certificate.
How we fixed it:
To mitigate the issue the cache layer was removed as fast as possible. After the issue was later addressed, the cache layer was reinstated the next day on Oct 1st.
What we are doing to prevent it from happening again:
We’re adding additional monitoring checks to detect any issues with the caching layer.
Resolved
We have taken remedial action and have seen tests pass as intended. All services are fully operational.
Update
We continue to investigate the certificate errors and mitigate its impact.
Investigating
Screener tests using ngrok might get an invalid certificate error when running. We are investigating.
This incident affected: Visual Testing (Visual Testing Hub, Visual Testing Infrastructure).