On Friday November 9th between 5:20 pm and 6:30 pm PST, wait times for tests to start were high. One of our services ran out of memory, which caused a cascade of problems in other services. Eventually this resulted in high wait times. We manually restarted the affected service. We’ve adjusted our memory monitoring tools to alert us well before we are in danger of running out of memory. All services are now fully operational.
Posted about 1 month ago. Nov 09, 2018 - 18:57 PST