We are continuing to work on a fix for this issue.
Posted Jul 10, 2024 - 15:15 PDT
Identified
At approximately 11:00 AM CT, The DAL data center proactively transferred to generator power on a ATS3 due of excessive temperatures on one of the upstream utility transformers. The transfer was successful. After further investigation of the transformer, insulation damage showing exposed electrical conductors was discovered and Oncor Electric, our upstream utility provider disconnected the utility feed to enable further investigation of the transformer. At approximately 1:58 PM CT, the generator servicing ATS 3 had a fan belt failure. As a result, any non-redundant, single corded critical loads associated with UPS #10 and #12 ran out of battery reserve at approximately 2:24 PM CT. Generator and Electrical vendors are actively engaged. A spare generator fan belt is on site and the generator vendor is currently working to replace the failed component. MTTR on the belt replacement is approximately 3:30 PM CT. Due to the utility and subsequent generator failure, temperatures in the data center will increase. The local team is currently working to re-feed HVAC units, associated with ATS 3, from an alternate power source within the facility. The HVAC refeed process should be complete by 3:45 PM CT.