[outages] Rightnow reporting several lines doen in bay area

Fri Apr 20 17:08:56 EDT 2012

Here is what I see on Level 3's ticketing system...

Network Event Summary: A partial fiber cut in Tustin, CA impacted transport unprotected, IP and Voice traffic. 911 calls were not impacted.
Event Ticket ID: 5539336
Market Area Affected: Tustin, CA
Ticket Create Date: 4/19/12 3:22:48 PM GMT
Impacted For: 12 hours 52 minutes
Event Status: Restored
Resolve Date: 4/20/12 3:55:14 PM GMT
Time Since Last Update: 15 hours 5 minutes

4/20/12 6:00:36 AM GMT  Technical service center confirm all customer services have been restored.

4/20/12 5:30:05 AM GMT  Transport NOC has completed system roll and all work has been completed. Technical service center is confirming service restoral at this time.

4/20/12 5:06:37 AM GMT  The fiber repair crew has advised they have closed all tubes as splicing is completed. Transport NOC is seeing some alarms on systems they report are operating outside of span. The Transport NOC is working with the field technician to roll those systems back to original fibers now that they have completed splicing. Technical service center is awaiting Transport NOC to complete roll before they can verify service restoral with the customers.

4/20/12 4:06:23 AM GMT  The fiber repair team report all splicing has been completed. Technical service center is confirming service restoral with the customer at this time.

4/20/12 3:42:14 AM GMT  The fiber repair team advised that splicing thru 132 is now complete and they only have two more tubes remaining to splice.

4/20/12 3:02:55 AM GMT  The fiber repair team advised that splicing in the south pit is complete. The north pit is complete thru 72 and crews are working on 73 thru 96 at this time.

4/20/12 2:07:47 AM GMT  The fiber repair team has advised a second splice crew has arrived to help expedite the fiber repair.

4/20/12 1:28:50 AM GMT  The fiber repair team has advised that they have completed splicing on over 48 fibers. The fiber restoration process is ongoing at this time and continues to be on schedule.

4/20/12 12:38:26 AM GMT  The Transport NOC has advised that the fiber repair team has begun splicing onsite. The Transport NOC has confirmed that the fiber repair team has spliced 24 fibers, which has restored services to some customers. The technicians successfully re-routed traffic and restored services to as many customers as possible. The splicing process is on schedule at this time.

4/19/12 11:33:04 PM GMT  The fiber repair team has advised that the fiber restoration strategy is to repair the fiber buffer tube by buffer tube in order. The fiber repair team is prepping the fiber for repair at this time and has updated the ETTR from 03:00 GMT to 06:00 GMT. The Transport NOC is investigating the possibility of re-routing customer traffic to expedite service restoration.

4/19/12 10:31:07 PM GMT  The Transport NOC has advised that some customer whose services were supposed to restore when traffic was rolled to an alternate path have been confirmed as still down. The Transport NOC has advised the fiber repair team to begin the fiber repair to expedite the restoration of services for all of the affected customers. The fiber repair team has provided and ETR of 4 Hours.

4/19/12 9:26:18 PM GMT  The Transport NOC advised that all systems have been rolled to an alternate path with the exception of some Dark Fiber customers. The Transport NOC advised that the Dark Fiber customers that are up could experience an additional impact to services once the fiber repair commences. The Technical Service Center is compiling a list of Dark Fiber customers that could be impacted so that they can be notified of the situation. The Splicing crew has the damaged fibers excavated and is in the process of placing new cable in the fiber ducks that were not damaged.

4/19/12 8:10:45 PM GMT  The Transport NOC advised that protected traffic was restored as of 19:18 GMT. The Global Field Service technicians continue to work to roll the remaining impacted traffic to an alternate path. The splicing crew is onsite still awaiting the arrival of the digging equipment; however, no ETA was available at this time.

4/19/12 7:05:50 PM GMT  The Transport NOC advises that three systems have been rolled and alarms have begun to clear at this time. Global Field Services is in the process of rolling the fourth system and all six systems should be rolled and restored by 20:00 GMT; the Transport NOC continues to monitor alarms. Global Field Services has reported that underground crews are onsite at the cut site and digging equipment should arrive to the site at approximately 19:45 GMT. The Technical Service Center reported that protected customer services are being impacted due to this issue. The Transport NOC confirmed this and has upgraded the event to a transport protected outage.

4/19/12 6:09:51 PM GMT  The Transport NOC advised that the Global Field Service technicians will commence to roll traffic to an alternate route in approximately 15-minutes. The fiber splicing crew is expected to be onsite at the damage location in approximately 30-minutes. No estimated time to restore is available at this time.

4/19/12 5:08:34 PM GMT  The Transport NOC advises that splice crews are en-route to the site at this time with an ETA of 17:45 GMT. Global Field Service Technicians are in the process of rolling services to spare fibers as temporary repair and advise that services should be migrated and restored shortly. The Transport NOC is monitoring alarms and will advise when all associated alarms have cleared and services are restored.

4/19/12 4:39:07 PM GMT  The Transport NOC advises that the Field Technician arrived to the site and shot the fiber, and is seeing a partial fiber cut at approximately 42 km from the Tustin, CA site. Global Field Services advises that construction crews have been engaged and are en-route to the site with an ETA of 17:00 GMT. Splice crews have been engaged to dispatch to the site for investigation; no ETA has is available at this time. A Field Technician has reported that the partial cut appears to have been caused by a billboard crew who was boring in the area. The Transport NOC states that roll options are also being investigated at this time as there are 3 fibers remaining in the 156-count fiber cable to use to roll high-priority services too. No estimated time to complete the service roll has been provided. The Transport NOC confirms that IP and Voice services are also being impacted due to this issue and has upgraded this event to critical level. No estimated time to restore is available.

4/19/12 3:25:02 PM GMT  The Transport NOC responded to loss of signal alarms, indicating a fiber cut in the Tustin, CA area, which is impacting unprotected traffic. Field Services have been engaged to dispatch to the site to shoot the fibers for further isolation. Estimated time of arrival to the site is 60 minutes.

Regards,
Tim

From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On Behalf Of Booher, Mike
Sent: Thursday, April 19, 2012 4:48 PM
To: outages at outages.org
Subject: Re: [outages] Rightnow reporting several lines doen in bay area

partial fiber cut at approximately 42 km from the Tustin, CA
site.   It has been reported that the partial cut appears to have been caused by a
billboard crew who was boring in the area.  The splicing crew is onsite still awaiting the arrival of the digging equipment;
however, no ETA was available at this time

Thanks!

Mike

From: outages-bounces at outages.org<mailto:outages-bounces at outages.org> [mailto:outages-bounces at outages.org]<mailto:[mailto:outages-bounces at outages.org]> On Behalf Of Blake Pfankuch
Sent: Thursday, April 19, 2012 16:32
To: Martin Hepworth; Owen DeLong
Cc: outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] Rightnow reporting several lines doen in bay area

Sounds like based on time, this could have been a cause of the major cBeyond issues this morning too.

Blake Pfankuch

From: outages-bounces at outages.org<mailto:outages-bounces at outages.org> [mailto:outages-bounces at outages.org]<mailto:[mailto:outages-bounces at outages.org]> On Behalf Of Martin Hepworth
Sent: Thursday, April 19, 2012 2:27 PM
To: Owen DeLong
Cc: outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] Rightnow reporting several lines doen in bay area

Looking at another report here looks like a fair amount if the western usa is affected by this

--
Martin

On Thursday, 19 April 2012, Owen DeLong wrote:
When did LA<->Dallas become part of the Bay Area?

Owen

On Apr 19, 2012, at 1:06 PM, <Vinny_Abello at dell.com<javascript:;>> wrote:

> There is a large fiber cut in Los Angeles affecting many providers at this time that's been going on since earlier today. I'm also being affected by this between LA and Dallas and seeing latency on other networks in the LA area as a result of congestion due to decreased capacity.
>
> -Vinny
>
> -----Original Message-----
> From: outages-bounces at outages.org<javascript:;> [mailto:outages-bounces at outages.org<javascript:;>] On Behalf Of Anthony Ledesma
> Sent: Thursday, April 19, 2012 3:52 PM
> To: Martin Hepworth; outages at outages.org<javascript:;>
> Subject: Re: [outages] Rightnow reporting several lines doen in bay area
>
> We have received a report from Zuora stating the following:
>
>> We have confirmed with our datacenter that there was a fiber line cut between Dallas and LA which is highly affecting Internet traffic.
>>
>> We will continue to work with our datacenter and provide updates.  We will receive an estimate when it will be repaired in about 2 hours.
>>
>> We appreciate your patience with the Internet.
>
>
> --
> Anthony Ledesma | FireHost
>
> From: outages-bounces at outages.org<javascript:;> [mailto:outages-bounces at outages.org<javascript:;>] On Behalf Of Martin Hepworth
> Sent: Thursday, April 19, 2012 12:43 PM
> To: outages at outages.org<javascript:;>
> Subject: [outages] Rightnow reporting several lines doen in bay area
>
> Anyone got any further info, they are pointing the finger at their providers
>
> --
> Martin
>
>
> --
> --
> Martin Hepworth
> Oxford, UK
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org<javascript:;>
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org<javascript:;>
> https://puck.nether.net/mailman/listinfo/outages

--
--
Martin Hepworth
Oxford, UK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20120420/6e1043db/attachment.htm>