[outages] Level 3 issues in Ashburn

Fri Jan 17 12:21:44 EST 2014

here is the final outage info. it has been cleared

1/17/14 10:46:07 AM GMTThe Technical Services Center has confirmed that all
services are up and active. If additional issues are experienced, please
contact the Level 3 Technical Service Center at 1-877-4LEVEL3.1/17/14
10:36:53 AM GMTThe Technical Services Center is still confirming the
restoration of all customers.1/17/14 9:17:13 AM GMTThe Technical Services
Center is still confirming the restoration of all customers.1/17/14 8:17:03
AM GMTTransport NOC reports splicing is complete. The technical service
center has been engaged to confirm restoration with customers.1/17/14
7:52:36 AM GMTField crews report splicing continues and there are 2 tubes
remaining to be spliced. Transport NOC is monitoring alarms.1/17/14 6:58:51
AM GMTTransport NOC reports field teams continue splicing and are reporting
6 tubes remaining to be spliced. An ETTR of 1 hour has been provided.1/17/14
5:46:03 AM GMTTransport NOC advises the field crews have completed splicing
in the West pit and have 8 tubes remaining in the East pit. Teams continue
splicing and expect to be complete in 1 hour.1/17/14 5:04:06 AM GMTTransport
NOC advises field crews are reporting splicing is just over half way
complete. It is estimated that splicing should be complete in about 2 hours
or so.1/17/14 4:31:56 AM GMTTransport NOC reports splicing continues in the
field. The West pit reports 18 tubes spliced while the East is reporting 12
tubes spliced. Alarms are clearing as splicing continues.1/17/14 3:34:35 AM
GMTTransport NOC reports the field advises they are almost half way
complete with splicing and expect the remaining splicing to be complete in
2 to 3 hours. Transport NOC is monitoring alarms as they clear.

.....................................
*Luke Rockwell*
Systems and Support Analyst
Information Technology

.....................................

Cal Alumni Association | UC Berkeley
1 Alumni House, Berkeley, CA 94720
*T* 510.900.8196
*F* 510.642.6252

.....................................

*Commitment, Support, Passion*

_____________________________________
*alumni.berkeley.edu <http://alumni.berkeley.edu/>*

_____________________________________
*Facebook <https://www.facebook.com/CalAlumni> | LinkedIn
<http://www.linkedin.com/groups?gid=70245>*

On Thu, Jan 16, 2014 at 7:25 PM, Luke Rockwell <
luke.rockwell at alumni.berkeley.edu> wrote:

> 1/17/14 2:51:29 AM GMT The Transport NOC informed that splicing is
> ongoing to restore services and as splicing progresses, alarms are
> continuing to clear. The estimated restoral time of 05:00 GMT remains
> unchanged at this time. 1/17/14 1:19:44 AM GMT Splicing has commenced on
> the damaged fiber and as splicing progresses, alarms are beginning to
> clear. It has been prevised that this splice work will be extensive and an
> estimated restoral time of 05:00 GMT has been provided.
>
> .....................................
> *Luke Rockwell*
> Systems and Support Analyst
> Information Technology
>
> .....................................
>
> Cal Alumni Association | UC Berkeley
> 1 Alumni House, Berkeley, CA 94720
> *T* 510.900.8196
> *F* 510.642.6252
>
> .....................................
>
> *Commitment, Support, Passion*
>
> _____________________________________
> *alumni.berkeley.edu <http://alumni.berkeley.edu/>*
>
> _____________________________________
> *Facebook <https://www.facebook.com/CalAlumni> | LinkedIn
> <http://www.linkedin.com/groups?gid=70245>*
>
>
> On Thu, Jan 16, 2014 at 5:20 PM, Luke Rockwell <
> luke.rockwell at alumni.berkeley.edu> wrote:
>
>> 1/17/14 1:19:44 AM GMT Splicing has commenced on the damaged fiber and
>> as splicing progresses, alarms are beginning to clear. It has been prevised
>> that this splice work will be extensive and an estimated restoral time of
>> 05:00 GMT has been provided.
>>
>> .....................................
>> *Luke Rockwell*
>> Systems and Support Analyst
>> Information Technology
>>
>> .....................................
>>
>> Cal Alumni Association | UC Berkeley
>> 1 Alumni House, Berkeley, CA 94720
>> *T* 510.900.8196
>> *F* 510.642.6252
>>
>> .....................................
>>
>> *Commitment, Support, Passion*
>>
>> _____________________________________
>> *alumni.berkeley.edu <http://alumni.berkeley.edu/>*
>>
>> _____________________________________
>> *Facebook <https://www.facebook.com/CalAlumni> | LinkedIn
>> <http://www.linkedin.com/groups?gid=70245>*
>>
>>
>> On Thu, Jan 16, 2014 at 3:44 PM, Luke Rockwell <
>> luke.rockwell at alumni.berkeley.edu> wrote:
>>
>>> 1/16/14 11:41:42 PM GMT Field Services have completed preparation work
>>> for splicing on one end, but the other end is still being prepped as
>>> unforeseen issues caused delays. It has been informed that splicing is
>>> expected to begin at 00:30 GMT.
>>>
>>>
>>> On Thu, Jan 16, 2014 at 3:31 PM, Tim Glen <tglen at asicentral.com> wrote:
>>>
>>>>  1/16/14 10:52:27 PM GMT   (5:52:27 PM EST)
>>>>
>>>> The Transport NOC informed that Field Services are continuing their
>>>> splicing preparations, which are expected to be completed by 00:00 GMT (7:00
>>>> PM EST) and splicing will begin subsequently.
>>>>
>>>>
>>>>
>>>>
>>>>  ------------------------------
>>>> *From:* Outages [mailto:outages-bounces at outages.org] *On Behalf Of *Luke
>>>> Rockwell
>>>> *Sent:* Thursday, January 16, 2014 4:46 PM
>>>>
>>>> *To:* outages at outages.org
>>>> *Subject:* Re: [outages] Level 3 issues in Ashburn
>>>>
>>>>    1/16/14 9:44:58 PM GMT The Transport NOC informed that Field
>>>> Services are anticipating splicing preparations will begin at approximately
>>>> 22:00 GMT. It has been informed that this process will take approximately
>>>> two hours to complete, and splicing will commence thereafter.
>>>>
>>>>
>>>> On Thu, Jan 16, 2014 at 1:29 PM, Tim Glen <tglen at asicentral.com> wrote:
>>>>
>>>>>  1/16/14 9:05:48 PM GMT
>>>>>
>>>>> The Transport NOC advised that Field Services are currently working to
>>>>> replace the damaged conduit so that the replacement fiber can be placed in.
>>>>> An estimated start time for splicing is unavailable at this time.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *From:* Outages [mailto:outages-bounces at outages.org] *On Behalf Of *Luke
>>>>> Rockwell
>>>>> *Sent:* Thursday, January 16, 2014 3:07 PM
>>>>>
>>>>> *To:* outages at outages.org
>>>>> *Subject:* Re: [outages] Level 3 issues in Ashburn
>>>>>
>>>>>
>>>>>
>>>>> Ouch
>>>>>
>>>>>
>>>>>
>>>>> 1/16/14 8:00:49 PM GMT
>>>>>
>>>>> Field Services are on site performing excavation of the impacted
>>>>> fiber, which is fairly deep. When the boring crew hit the fiber,
>>>>> approximately 25 feet of fiber was pulled out and will need to be repaired.
>>>>> Restoration efforts are underway, but no estimated restoral time is
>>>>> currently available.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jan 16, 2014 at 11:12 AM, Eric Dugas <EDugas at zerofail.com>
>>>>> wrote:
>>>>>
>>>>> Also affects our Hibernia Atlantic transport to NYC. I've received the
>>>>> following update at 13:22 EST:
>>>>>
>>>>> "Please be advised that Hibernia Networks is currently experiencing a
>>>>> network outage between Ashburn and Pittsburgh, USA since 16:40 GMT. We are
>>>>> currently investigating the issue.
>>>>>
>>>>> Update1 - Please be informed that, our partner is currently
>>>>> investigating the issue and has dispatched techs to site, there is
>>>>> currently no ETA available, further updates to follow."
>>>>>
>>>>> Eric Dugas
>>>>> edugas at zerofail.com
>>>>> Network Management Group
>>>>> ZEROFAIL / AS40191
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20140117/13c9d26f/attachment.htm>