Update:
At 1:45 PM PT, Power was restored to the site and machines had been started to be powered back on. The datacenter is back on full utility power with UPS backup. Datacenter engineers are continuing to look into why the generator supplied power at a voltage outside of expected range for the UPS which caused the UPS to not switch over.

Original issue:

Our datacenter US-NC-1 has reported a power failure at the US-NC-1 facility that occurred at approximately 8:00 AM PT on April 4th.

This failure has taken down the entire A-side power. The servers currently failing are not N+N Redundant; they are N+1. Typically, in a power outage, UPS and Generators act as backup, however, this failover is not working.

Onsite teams are working on providing a Root Cause Analysis (RCA) to find out why the backup power was unable to kick on to run these servers.

Updates will be shared as our datacenter partners provide estimated repair times and as more information becomes available.