[outages] Explosion in Nashville

Sat Dec 26 22:44:48 EST 2020

Apologies if this has been already posted.

8pm update:

All - Teams have completed their walk-through with the Fire Marshal and
local authorities. Officials have given teams approval to begin restoring
power to the 2nd and 3rd floors. After the 2nd and 3rd floors are
energized, teams will begin working to restore power to the 1st, 4th and
5th floors. A fuel truck was brought in to refill the fuel tanks on the
generators and a Caterpilar technician is onsite getting the generators
ready to run. Teams will start one generator at a time. Once generators are
running, teams can begin energizing power plants and charging batteries.
Many batteries have reversed polarity and it is unclear at this time, if
those batteries can be recharged. AT&T Structural Engineer has found no
concrete or steel damage that would impair restoring power and service in
the building that caused major concern.

Core CBB teams continue to work on the secondary plan to reroute around the
Nashville 2nd Ave location. Service in and around the Lexington/Winchester,
KY has returned for 3G and 4G service when CBB service was rerouted around
the Nashville 2nd Ave location. 3G Data service is still degraded within
that specific area. Work will continue with the CBB reroute plans across
several locations around the Southeast Region including the Nashville, MTSO
(Brick Church Pike).

Two SATCOLTs are positioned on the East and West sides of the NSVLTNMT
Central Office and both are on-air and taking traffic to aid in
communication. There are currently 6 assets on air – the 2 at the CO plus 4
that are supporting FN Customer Requests. There are 4 assets on route to
their respective deployment locations- 2 in Alabama and 2 in TN. One asset
has arrived at its intended deployment location in TN and is in the process
of being turned up. Another 3 requests have been prioritized and assets
that just arrived at the staging location are being deployed to their
target locations in TN. Issues remain at various locations in GA, KY, & MS.

On Sat, Dec 26, 2020 at 6:08 PM Jay Ashworth via Outages <
outages at outages.org> wrote:

> It is. So unless that message that you reposted came to you with a flag
> line suggesting it should not be disseminated, you're welcome to post it
> here.
>
>
> Cheers,
> -- jr '<admin/>' a
>
> On December 26, 2020 1:35:29 PM EST, Ross Tajvar via Outages <
> outages at outages.org> wrote:
>>
>> Sorry I don't follow. Isn't the point of this list to keep people
>> informed?
>>
>> On Sat, Dec 26, 2020, 1:31 PM Luis Diaz <noc at riotgames.com> wrote:
>>
>>> May want to consider moving this discussion to something more private.
>>>
>>>
>>>
>>> On Sat, Dec 26, 2020 at 9:30 AM Ross Tajvar via Outages <
>>> outages at outages.org> wrote:
>>>
>>>> Latest update from AT&T:
>>>>
>>>>
>>>>
>>>> AT&T Corporate Real Estate teams are now gathering on site after
>>>> pausing work last night due a fire that reignited on site around midnight.
>>>> The Fire Marshall has joined the AT&T team to provide official approval to
>>>> proceed. Last night, the teams were able to accomplish testing the busbar,
>>>> placing the temporary generators, completing power cabling, and opening all
>>>> the breakers on the individual floors. It is estimated that this round of
>>>> power related work is about 75% complete. The target is for energizing is
>>>> 1200 CST. A partial analysis by the AT&T Structural Engineer initial found
>>>> no concrete or steel damage that caused major concern. Once team re-engage
>>>> this morning, they will complete their full analysis and ensure it is safe
>>>> for restoration crews to proceed. AT&T Mechanical Engineers have confirmed
>>>> there is roughly 3 feet of water in the basement with potential damage to
>>>> mechanical equipment in that area. It is also suspected that commercial
>>>> power may not be available for some time and as a result power will come
>>>> from AT&T and Tennessee Emergency Management Agency generators. Additional
>>>> technician support will be made available rotating over a 24 hour period.
>>>>
>>>> Core CBB teams continue to work on the secondary plan to reroute around
>>>> the Nashville 2nd Ave location. They had some success this morning at 05:05
>>>> CT when core connectivity was established for the Winchester, KY office.
>>>> Partial 911 connectivity has been restored in that specific area. Work will
>>>> continue on the CBB reroute plans in several locations around the Southeast
>>>> Region.
>>>>
>>>> We have installed a SatCOLT positioned in the area of the CO NSVLTNMT
>>>> building and is up and taking traffic to aid in communication. The
>>>> secondary SatCOLT arrived and has been turned up to provide additional
>>>> coverage.
>>>>
>>>> The next update will be issued at approximately 13:00 CT, unless there
>>>> is a significant change in status
>>>>
>>>> On Fri, Dec 25, 2020, 7:49 PM Ross Tajvar <ross at tajvar.io> wrote:
>>>>
>>>>> Got this internally, sharing here:
>>>>>
>>>>> Here is the update from AT&T:
>>>>> >
>>>>> > The front 1/3rd of their building is structuraly damaged.  That
>>>>> includes fiber entry.
>>>>> > They have no power and there is major flooding on all floors
>>>>> including the basement.
>>>>> > The Fire Department is en route with pump trucks to help with the
>>>>> flooding.
>>>>> > They have Building Engineers and Electricians on the way to  look
>>>>> into the structure and getting power turned on.
>>>>> > They have 2 850KW generators on the way and 2 1MW generators on
>>>>> standby.
>>>>> > They have new A/C Chillers on the way (Existing Chiller plant was
>>>>> destroyed)
>>>>>
>>>>> On Fri, Dec 25, 2020 at 3:38 PM Jay R. Ashworth via Outages <
>>>>> outages at outages.org> wrote:
>>>>>
>>>>>> ----- Original Message -----
>>>>>> > From: "Kurt Duncan via Outages" <outages at outages.org>
>>>>>>
>>>>>> > Anyone else seeing phone or internet issues due to the AT&T
>>>>>> building being
>>>>>> > damaged by the explosion in Nashville ?
>>>>>>
>>>>>> Let's please continue to confine replies here to specific information
>>>>>> about
>>>>>> network impacts, and keep the speculation threads to -discuss; thanks
>>>>>> to
>>>>>> whomever already started that thread.  I would name them, but the
>>>>>> caffeine
>>>>>> hasn't kicked in yet.
>>>>>>
>>>>>> If you're in that building in Nash, we'll be particularly interested
>>>>>> in
>>>>>> anything you're allowed to tell us in between bailing.
>>>>>>
>>>>>> Cheers,
>>>>>> -- jr '<admin/>' a
>>>>>> --
>>>>>> Jay R. Ashworth                  Baylink
>>>>>> jra at baylink.com
>>>>>> Designer                     The Things I Think
>>>>>>  RFC 2100
>>>>>> Ashworth & Associates       http://www.bcp38.info          2000 Land
>>>>>> Rover DII
>>>>>> St Petersburg FL USA      BCP38: Ask For It By Name!           +1 727
>>>>>> 647 1274
>>>>>> _______________________________________________
>>>>>> Outages mailing list
>>>>>> Outages at outages.org
>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>
>>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-- 

*Walt Amason* | Director of Information Technology | Kelley Companies
*cell* (601) 382-5888 | *office* (601) 735-2541 | *website*
www.kelleycompanies.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20201226/6453e590/attachment.htm>