[outages] Explosion in Nashville

Ross Tajvar ross at tajvar.io
Sun Dec 27 02:18:38 EST 2020 

This is from 11pm. Some duplicate info here, but some other stuff as well:

Latest AT&T Update:



Within the Nashville Central Office, the 2nd, 3rd, 4th and 6th floors
currently have power to the rectifiers.


The 6th floor rectifiers are currently being energized to charge batteries.



Power has been applied to the Air Conditioning risers, and Gensets are
turned up. One handler is turned up on each of floors 2 and 3.



Pumping of the three feet of water in the basement has been completed;
however, the FBI has not yet released this portion of the building back to
AT&T.

It has been confirmed that the four utility service feeds are offline, and
the utility transformer bus duct experienced damage.

On Sat, Dec 26, 2020, 10:45 PM Walt Amason <wamason at kelleycompanies.com>
wrote:

> Apologies if this has been already posted.
>
>
> 8pm update:
>
> All - Teams have completed their walk-through with the Fire Marshal and
> local authorities. Officials have given teams approval to begin restoring
> power to the 2nd and 3rd floors. After the 2nd and 3rd floors are
> energized, teams will begin working to restore power to the 1st, 4th and
> 5th floors. A fuel truck was brought in to refill the fuel tanks on the
> generators and a Caterpilar technician is onsite getting the generators
> ready to run. Teams will start one generator at a time. Once generators are
> running, teams can begin energizing power plants and charging batteries.
> Many batteries have reversed polarity and it is unclear at this time, if
> those batteries can be recharged. AT&T Structural Engineer has found no
> concrete or steel damage that would impair restoring power and service in
> the building that caused major concern.
>
> Core CBB teams continue to work on the secondary plan to reroute around
> the Nashville 2nd Ave location. Service in and around the
> Lexington/Winchester, KY has returned for 3G and 4G service when CBB
> service was rerouted around the Nashville 2nd Ave location. 3G Data service
> is still degraded within that specific area. Work will continue with the
> CBB reroute plans across several locations around the Southeast Region
> including the Nashville, MTSO (Brick Church Pike).
>
> Two SATCOLTs are positioned on the East and West sides of the NSVLTNMT
> Central Office and both are on-air and taking traffic to aid in
> communication. There are currently 6 assets on air – the 2 at the CO plus 4
> that are supporting FN Customer Requests. There are 4 assets on route to
> their respective deployment locations- 2 in Alabama and 2 in TN. One asset
> has arrived at its intended deployment location in TN and is in the process
> of being turned up. Another 3 requests have been prioritized and assets
> that just arrived at the staging location are being deployed to their
> target locations in TN. Issues remain at various locations in GA, KY, & MS.
>
>
>
>
> On Sat, Dec 26, 2020 at 6:08 PM Jay Ashworth via Outages <
> outages at outages.org> wrote:
>
>> It is. So unless that message that you reposted came to you with a flag
>> line suggesting it should not be disseminated, you're welcome to post it
>> here.
>>
>>
>> Cheers,
>> -- jr '<admin/>' a
>>
>> On December 26, 2020 1:35:29 PM EST, Ross Tajvar via Outages <
>> outages at outages.org> wrote:
>>>
>>> Sorry I don't follow. Isn't the point of this list to keep people
>>> informed?
>>>
>>> On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc at riotgames.com> wrote:
>>>
>>>> May want to consider moving this discussion to something more private.
>>>>
>>>>
>>>>
>>>> On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <
>>>> outages at outages.org> wrote:
>>>>
>>>>> Latest update from AT&T:
>>>>>
>>>>>
>>>>>
>>>>> AT&T Corporate Real Estate teams are now gathering on site after
>>>>> pausing work last night due a fire that reignited on site around midnight.
>>>>> The Fire Marshall has joined the AT&T team to provide official approval to
>>>>> proceed. Last night, the teams were able to accomplish testing the busbar,
>>>>> placing the temporary generators, completing power cabling, and opening all
>>>>> the breakers on the individual floors. It is estimated that this round of
>>>>> power related work is about 75% complete. The target is for energizing is
>>>>> 1200 CST. A partial analysis by the AT&T Structural Engineer initial found
>>>>> no concrete or steel damage that caused major concern. Once team re-engage
>>>>> this morning, they will complete their full analysis and ensure it is safe
>>>>> for restoration crews to proceed. AT&T Mechanical Engineers have confirmed
>>>>> there is roughly 3 feet of water in the basement with potential damage to
>>>>> mechanical equipment in that area. It is also suspected that commercial
>>>>> power may not be available for some time and as a result power will come
>>>>> from AT&T and Tennessee Emergency Management Agency generators. Additional
>>>>> technician support will be made available rotating over a 24 hour period.
>>>>>
>>>>> Core CBB teams continue to work on the secondary plan to reroute
>>>>> around the Nashville 2nd Ave location. They had some success this morning
>>>>> at 05:05 CT when core connectivity was established for the Winchester, KY
>>>>> office. Partial 911 connectivity has been restored in that specific area.
>>>>> Work will continue on the CBB reroute plans in several locations around the
>>>>> Southeast Region.
>>>>>
>>>>> We have installed a SatCOLT positioned in the area of the CO NSVLTNMT
>>>>> building and is up and taking traffic to aid in communication. The
>>>>> secondary SatCOLT arrived and has been turned up to provide additional
>>>>> coverage.
>>>>>
>>>>> The next update will be issued at approximately 13:00 CT, unless there
>>>>> is a significant change in status
>>>>>
>>>>> On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross at tajvar.io> wrote:
>>>>>
>>>>>> Got this internally, sharing here:
>>>>>>
>>>>>> Here is the update from AT&T:
>>>>>> >
>>>>>> > The front 1/3rd of their building is structuraly damaged.  That
>>>>>> includes fiber entry.
>>>>>> > They have no power and there is major flooding on all floors
>>>>>> including the basement.
>>>>>> > The Fire Department is en route with pump trucks to help with the
>>>>>> flooding.
>>>>>> > They have Building Engineers and Electricians on the way to  look
>>>>>> into the structure and getting power turned on.
>>>>>> > They have 2 850KW generators on the way and 2 1MW generators on
>>>>>> standby.
>>>>>> > They have new A/C Chillers on the way (Existing Chiller plant was
>>>>>> destroyed)
>>>>>>
>>>>>> On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
>>>>>> outages at outages.org> wrote:
>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> > From: "Kurt Duncan via Outages" <outages at outages.org>
>>>>>>>
>>>>>>> > Anyone else seeing phone or internet issues due to the AT&T
>>>>>>> building being
>>>>>>> > damaged by the explosion in Nashville ?
>>>>>>>
>>>>>>> Let's please continue to confine replies here to specific
>>>>>>> information about
>>>>>>> network impacts, and keep the speculation threads to -discuss;
>>>>>>> thanks to
>>>>>>> whomever already started that thread.  I would name them, but the
>>>>>>> caffeine
>>>>>>> hasn't kicked in yet.
>>>>>>>
>>>>>>> If you're in that building in Nash, we'll be particularly interested
>>>>>>> in
>>>>>>> anything you're allowed to tell us in between bailing.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> -- jr '<admin/>' a
>>>>>>> --
>>>>>>> Jay R. Ashworth                  Baylink
>>>>>>> jra at baylink.com
>>>>>>> Designer                     The Things I Think
>>>>>>>  RFC 2100
>>>>>>> Ashworth & Associates       http://www.bcp38.info          2000
>>>>>>> Land Rover DII
>>>>>>> St Petersburg FL USA      BCP38: Ask For It By Name!           +1
>>>>>>> 727 647 1274
>>>>>>> _______________________________________________
>>>>>>> Outages mailing list
>>>>>>> Outages at outages.org
>>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>>
>>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> --
>
>
> *Walt Amason* | Director of Information Technology | Kelley Companies
> *cell* (601) 382-5888 | *office* (601) 735-2541 | *website*
> www.kelleycompanies.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20201227/9f1fdf2b/attachment.htm>

More information about the Outages mailing list