[outages] Atlanta 56 Marietta - MMR Power issue

Wed Jul 12 09:41:00 EDT 2017

My biggest concern is we lost both A and B power. Which I would expect to be on separate "Power Grids / Infrastructure"

Here was the maintenance notice sent out.

July 12, 2017
MAINTENANCE NOTIFICATION - 15-MINUTE-REMINDER - MTST# 00443066 - ATL1

Valued Customer,

Digital Realty engineers will be performing maintenance to add capacity to existing PDUs (Power Distribution Units) at our ATL1 (56 Marietta Street, Atlanta, GA) facility on the dates/times shown below.  All work associated should be seamless and your systems should not see any loss of power.  As with any maintenance we do not expect any power outages, but there is always a chance that critical systems could be impacted during these procedures.  Please ensure that if you subscribe to redundant power sources that your equipment is properly distributed.

Maintenance TST# 00443066
Location: ATL1 (56 Marietta, Atlanta)
Floor(s) / Suite(s): Atlanta Sub-Basement;Atlanta Basement;Atlanta 1st FL;Atlanta 2nd FL;Atlanta 3rd FL;Atlanta 4th FL;Atlanta 7th FL;Atlanta 8th FL;Atlanta 9th FL;Atlanta 10th FL
Maintenance Type: Planned Maintenance
Priority: YELLOW Event - PSA (Potentially Service Affecting)
Product(s)/Asset(s) Affected: Electrical Distribution
Description: Critical Power Redistribution Phase 1-PDU Source Selector Feeds Upgrade
Start Date/Time: 07/12/2017 07:00 AM EDT
End Date/Time: 07/13/2017 07:00 PM EDT
Duration: 7am - 7pm Daily
Related Work Orders:
Related Circuit IDs:

Please refer any questions to:
The Global Command Center at gcc at digitalrealty.com or 888-835-9832

Erik Sundberg
Sr. Network Engineering
Network Engineering Department
p: 773.661.5532
c: 708.710.7419
e: esundberg at nitelusa.com
Main: 888.450.2100
NOC 24/7: 866.892.0915
350 North Orleans Street, Suite 1300N Chicago, IL 60654
www.nitelusa.com

Managed Telecom Services
MPLS | Ethernet | Private Line | Internet | Voice | Security

-----Original Message-----
From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Matt Hoppes via Outages
Sent: Wednesday, July 12, 2017 8:35 AM
To: Marty Strong
Cc: outages at outages.org
Subject: Re: [outages] Atlanta 56 Marietta - MMR Power issue

Ok. But why was there no battery backup?

> On Jul 12, 2017, at 9:33 AM, Marty Strong via Outages <outages at outages.org> wrote:
>
> Received this from DRT
>
>> RED Event - Generator Failure / Critical Load Impact
>> 56 Marietta Atlanta GA
>>
>> During scheduled maintenance, the engineering team at 56 Marietta reports that the generator failed resulting in a loss of critical IT load to equipment in the space. The site team is investigating the source of the issue and scope of the impact. Further updates will be provided as additional information becomes available.
>>
>> Thank you,
>> Digital Realty Operations
>
> Regards,
> Marty Strong
> --------------------------------------
> Cloudflare - AS13335
> Network Engineer
> marty at cloudflare.com
> +44 7584 906 055
> smartflare (Skype)
>
> https://www.peeringdb.com/asn/13335
>
>> On 12 Jul 2017, at 14:23, Jay Weekley via Outages <outages at outages.org> wrote:
>>
>> We came back up about a minute ago.
>>
>> NSC Global via Outages wrote:
>>>
>>> Looks like everything is coming back up now.
>>>
>>> No explanation just yet.
>>>
>>> *Andrey Leo*
>>>
>>> Network Analyst I
>>>
>>> *T(US)* *1-866-484-2588*   |   T(UK) *0800-840-7491*
>>>
>>>
>>>
>>> 30-31 Town Quay
>>> Southampton, Hampshire  SO14 2AQ UK
>>> *cogecopeer1.com* <http://www.cogecopeer1.com/>
>>>
>>> <http://www.cogecopeer1.com/>
>>>
>>> *From:*Outages [mailto:outages-bounces at outages.org] *On Behalf Of *Jamie Stephens via Outages
>>> *Sent:* 12 July 2017 14:16
>>> *To:* John Bergoon; outages at outages.org; Marty Strong
>>> *Subject:* Re: [outages] Atlanta 56 Marietta - MMR Power issue
>>>
>>> We are a provider on the 8th floor no power issues but logs show customers links down
>>>
>>> On Wed, Jul 12, 2017 at 9:13 AM John Bergoon via Outages <outages at outages.org <mailto:outages at outages.org>> wrote:
>>>
>>>   We also lost connectivity to some of our sip providers, our gear
>>>   is on the 7th floor not sure where they are.
>>>
>>>   Sent from my iPad
>>>
>>>
>>>   On Jul 12, 2017, at 8:11 AM, Marty Strong via Outages
>>>   <outages at outages.org <mailto:outages at outages.org>> wrote:
>>>
>>>       We lost something in Telx, but not in Equinix.
>>>
>>>       Device came back, but Telx IX ports still down.
>>>
>>>       Regards,
>>>       Marty Strong
>>>       --------------------------------------
>>>       Cloudflare - AS13335
>>>       Network Engineer
>>>       marty at cloudflare.com <mailto:marty at cloudflare.com>
>>>       +44 7584 906 055
>>>       smartflare (Skype)
>>>
>>>       https://www.peeringdb.com/asn/13335
>>>
>>>
>>>       On 12 Jul 2017, at 14:04, Chris Adams via Outages
>>>       <outages at outages.org <mailto:outages at outages.org>> wrote:
>>>
>>>           Once upon a time, Erich Kaiser via Outages
>>>           <outages at outages.org <mailto:outages at outages.org>> said:
>>>
>>>               Looks like there is a power issue in the DR/Telx 56
>>>               Marietta facility.
>>>
>>>               Just happened about 7:41 CST.
>>>
>>>           Our cabinet on the 6th floor appears to have lost power
>>>           (DC and AC)
>>>
>>>           briefly between 08:36:54 (last remote log message) and
>>>           08:38:58 (first
>>>
>>>           device booted) EDT.
>>>
>>>           --
>>>           Chris Adams <cma at cmadams.net <mailto:cma at cmadams.net>>
>>>
>>>           _______________________________________________
>>>
>>>           Outages mailing list
>>>
>>>           Outages at outages.org <mailto:Outages at outages.org>
>>>
>>>           https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>>>       _______________________________________________
>>>       Outages mailing list
>>>       Outages at outages.org <mailto:Outages at outages.org>
>>>       https://puck.nether.net/mailman/listinfo/outages
>>>
>>>   _______________________________________________
>>>   Outages mailing list
>>>   Outages at outages.org <mailto:Outages at outages.org>
>>>   https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>    Virus-free. www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>
>>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.