Saturday April 29th 2023, 07:03 - 09:06 UTC
Customers experienced elevated wait times and error rates for tests running on macOS and iOS simulators for the duration of the incident.
A third-party automated upgrade erroneously halted services supporting service discovery for one of our data centers. Without service discovery, VMs were not able to be restarted.
We restarted the service discovery services, and normal operation resumed.
We are preventing live upgrades of the service in question, moving it to our regular release workflow.