[ATL01] Power Outage
Incident Report for QuadraNet
Resolved
All repairs have been completed to our edge router and traffic has been gracefully shifted back to it without service interruption.
Posted Feb 18, 2022 - 14:40 PST
Monitoring
At this time we have restored network connectivity to our environment(s) and some customers have also been restored.

At this time we will begin the recovery of customers that are not currently restored. If you have an open (pending) ticket regarding this incident, we will review your request and work to restore your service (if not already restored).

The status of this situation is being changed at this time as QuadraNet has successfully deployed our disaster recovery playbook and we will continue efforts to restore customers who are still impacted by the original power failure.
Posted Feb 17, 2022 - 00:04 PST
Update
QuadraNet has not been successful at this time in restoring our edge routing equipment from the power failure.

At this time we will be exercising all available options in our disaster recovery plan in order to restore service to our customers.

Updates will be provided once they are available.
Posted Feb 16, 2022 - 23:01 PST
Update
At this time we have not been successful in restoring our routing equipment and we are still troubleshooting this with the datacenter engineers.
Posted Feb 16, 2022 - 22:09 PST
Update
At this time QuadraNet network engineers are working with local engineers to restore service to our routing equipment. Due to the hard power disconnect, it has resulted in the failure of several components in our edge routing equipment in this datacenter.

We will provide updates as they become available.
Posted Feb 16, 2022 - 21:26 PST
Update
Engineers have been provided access to the building after being cleared by the Atlanta Fire Department. We are currently working directly with the engineers onsite to restore service.
Posted Feb 16, 2022 - 19:37 PST
Update
The datacenter has informed us of the following:

"A contractor working on the 2nd floor of the 55 Marietta building cut into a water main and flooded the 2nd and 1st level of the building. The flooding extended to the generator room which took down the datacenter generator and ATS (Automatic Transfer Switch). The Atlanta Fire Department is currently on the scene and is not allowing access to the building until it has been cleared for safety."

QuadraNet is currently working diligently with our datacenter partner in the Atlanta location to restore service. We will provide additional updates as they become available.

Thanks,
Posted Feb 16, 2022 - 16:47 PST
Identified
The issue has been identified and a fix is being implemented.
Posted Feb 16, 2022 - 13:01 PST
Investigating
The datacenter at 55 Marietta has informed us of a power event that has impacted the entire datacenter floor. We are currently awaiting updates from the datacenter engineers.
Posted Feb 16, 2022 - 13:01 PST
This incident affected: ATL-C1.