<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#943634;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Here is what I see on Level 3’s ticketing system…<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Network Event Summary: A partial fiber cut in Tustin, CA impacted transport unprotected, IP and Voice traffic. 911 calls were not impacted. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Event Ticket ID: 5539336 <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Market Area Affected: Tustin, CA <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Ticket Create Date: 4/19/12 3:22:48 PM GMT <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Impacted For: 12 hours 52 minutes <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Event Status: Restored <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Resolve Date: 4/20/12 3:55:14 PM GMT <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Time Since Last Update: 15 hours 5 minutes <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/20/12 6:00:36 AM GMT Technical service center confirm all customer services have been restored. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/20/12 5:30:05 AM GMT Transport NOC has completed system roll and all work has been completed. Technical service center is confirming service restoral at this time. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/20/12 5:06:37 AM GMT The fiber repair crew has advised they have closed all tubes as splicing is completed. Transport NOC is seeing some alarms on systems they report are operating outside of span. The Transport NOC is working with the field technician to roll those systems back to original fibers now that they have completed splicing. Technical service center is awaiting Transport NOC to complete roll before they can verify service restoral with the customers. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/20/12 4:06:23 AM GMT The fiber repair team report all splicing has been completed. Technical service center is confirming service restoral with the customer at this time. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/20/12 3:42:14 AM GMT The fiber repair team advised that splicing thru 132 is now complete and they only have two more tubes remaining to splice. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/20/12 3:02:55 AM GMT The fiber repair team advised that splicing in the south pit is complete. The north pit is complete thru 72 and crews are working on 73 thru 96 at this time. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/20/12 2:07:47 AM GMT The fiber repair team has advised a second splice crew has arrived to help expedite the fiber repair. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/20/12 1:28:50 AM GMT The fiber repair team has advised that they have completed splicing on over 48 fibers. The fiber restoration process is ongoing at this time and continues to be on schedule. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/20/12 12:38:26 AM GMT The Transport NOC has advised that the fiber repair team has begun splicing onsite. The Transport NOC has confirmed that the fiber repair team has spliced 24 fibers, which has restored services to some customers. The technicians successfully re-routed traffic and restored services to as many customers as possible. The splicing process is on schedule at this time. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/19/12 11:33:04 PM GMT The fiber repair team has advised that the fiber restoration strategy is to repair the fiber buffer tube by buffer tube in order. The fiber repair team is prepping the fiber for repair at this time and has updated the ETTR from 03:00 GMT to 06:00 GMT. The Transport NOC is investigating the possibility of re-routing customer traffic to expedite service restoration. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/19/12 10:31:07 PM GMT The Transport NOC has advised that some customer whose services were supposed to restore when traffic was rolled to an alternate path have been confirmed as still down. The Transport NOC has advised the fiber repair team to begin the fiber repair to expedite the restoration of services for all of the affected customers. The fiber repair team has provided and ETR of 4 Hours. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/19/12 9:26:18 PM GMT The Transport NOC advised that all systems have been rolled to an alternate path with the exception of some Dark Fiber customers. The Transport NOC advised that the Dark Fiber customers that are up could experience an additional impact to services once the fiber repair commences. The Technical Service Center is compiling a list of Dark Fiber customers that could be impacted so that they can be notified of the situation. The Splicing crew has the damaged fibers excavated and is in the process of placing new cable in the fiber ducks that were not damaged. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/19/12 8:10:45 PM GMT The Transport NOC advised that protected traffic was restored as of 19:18 GMT. The Global Field Service technicians continue to work to roll the remaining impacted traffic to an alternate path. The splicing crew is onsite still awaiting the arrival of the digging equipment; however, no ETA was available at this time. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/19/12 7:05:50 PM GMT The Transport NOC advises that three systems have been rolled and alarms have begun to clear at this time. Global Field Services is in the process of rolling the fourth system and all six systems should be rolled and restored by 20:00 GMT; the Transport NOC continues to monitor alarms. Global Field Services has reported that underground crews are onsite at the cut site and digging equipment should arrive to the site at approximately 19:45 GMT. The Technical Service Center reported that protected customer services are being impacted due to this issue. The Transport NOC confirmed this and has upgraded the event to a transport protected outage. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/19/12 6:09:51 PM GMT The Transport NOC advised that the Global Field Service technicians will commence to roll traffic to an alternate route in approximately 15-minutes. The fiber splicing crew is expected to be onsite at the damage location in approximately 30-minutes. No estimated time to restore is available at this time. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/19/12 5:08:34 PM GMT The Transport NOC advises that splice crews are en-route to the site at this time with an ETA of 17:45 GMT. Global Field Service Technicians are in the process of rolling services to spare fibers as temporary repair and advise that services should be migrated and restored shortly. The Transport NOC is monitoring alarms and will advise when all associated alarms have cleared and services are restored. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/19/12 4:39:07 PM GMT The Transport NOC advises that the Field Technician arrived to the site and shot the fiber, and is seeing a partial fiber cut at approximately 42 km from the Tustin, CA site. Global Field Services advises that construction crews have been engaged and are en-route to the site with an ETA of 17:00 GMT. Splice crews have been engaged to dispatch to the site for investigation; no ETA has is available at this time. A Field Technician has reported that the partial cut appears to have been caused by a billboard crew who was boring in the area. The Transport NOC states that roll options are also being investigated at this time as there are 3 fibers remaining in the 156-count fiber cable to use to roll high-priority services too. No estimated time to complete the service roll has been provided. The Transport NOC confirms that IP and Voice services are also being impacted due to this issue and has upgraded this event to critical level. No estimated time to restore is available. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>4/19/12 3:25:02 PM GMT The Transport NOC responded to loss of signal alarms, indicating a fiber cut in the Tustin, CA area, which is impacting unprotected traffic. Field Services have been engaged to dispatch to the site to shoot the fibers for further isolation. Estimated time of arrival to the site is 60 minutes. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Regards,<o:p></o:p></span></p><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:10.0pt;font-family:"Verdana","sans-serif";color:#1F497D'>Tim<br><br><o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> outages-bounces@outages.org [mailto:outages-bounces@outages.org] <b>On Behalf Of </b>Booher, Mike<br><b>Sent:</b> Thursday, April 19, 2012 4:48 PM<br><b>To:</b> outages@outages.org<br><b>Subject:</b> Re: [outages] Rightnow reporting several lines doen in bay area<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>partial fiber cut at approximately 42 km from the Tustin, CA<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>site. It has been reported that the partial cut appears to have been caused by a<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>billboard crew who was boring in the area. The splicing crew is onsite still awaiting the arrival of the digging equipment;<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>however, no ETA was available at this time<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><p class=MsoNormal><span style='font-size:11.0pt;color:#1F497D'>Thanks!<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;color:#1F497D'>Mike </span><span style='font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a> <a href="mailto:[mailto:outages-bounces@outages.org]">[mailto:outages-bounces@outages.org]</a> <b>On Behalf Of </b>Blake Pfankuch<br><b>Sent:</b> Thursday, April 19, 2012 16:32<br><b>To:</b> Martin Hepworth; Owen DeLong<br><b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br><b>Subject:</b> Re: [outages] Rightnow reporting several lines doen in bay area<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#943634'>Sounds like based on time, this could have been a cause of the major cBeyond issues this morning too.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#943634'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:14.0pt;font-family:"Calibri","sans-serif";color:black'>Blake Pfankuch<o:p></o:p></span></b></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#943634'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a> <a href="mailto:[mailto:outages-bounces@outages.org]">[mailto:outages-bounces@outages.org]</a> <b>On Behalf Of </b>Martin Hepworth<br><b>Sent:</b> Thursday, April 19, 2012 2:27 PM<br><b>To:</b> Owen DeLong<br><b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br><b>Subject:</b> Re: [outages] Rightnow reporting several lines doen in bay area<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Looking at another report here looks like a fair amount if the western usa is affected by this<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>-- <o:p></o:p></p></div><div><p class=MsoNormal>Martin<br><br>On Thursday, 19 April 2012, Owen DeLong wrote:<o:p></o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'>When did LA<->Dallas become part of the Bay Area?<br><br>Owen<br><br>On Apr 19, 2012, at 1:06 PM, <<a href="javascript:;">Vinny_Abello@dell.com</a>> wrote:<br><br>> There is a large fiber cut in Los Angeles affecting many providers at this time that's been going on since earlier today. I'm also being affected by this between LA and Dallas and seeing latency on other networks in the LA area as a result of congestion due to decreased capacity.<br>><br>> -Vinny<br>><br>> -----Original Message-----<br>> From: <a href="javascript:;">outages-bounces@outages.org</a> [mailto:<a href="javascript:;">outages-bounces@outages.org</a>] On Behalf Of Anthony Ledesma<br>> Sent: Thursday, April 19, 2012 3:52 PM<br>> To: Martin Hepworth; <a href="javascript:;">outages@outages.org</a><br>> Subject: Re: [outages] Rightnow reporting several lines doen in bay area<br>><br>> We have received a report from Zuora stating the following:<br>><br>>> We have confirmed with our datacenter that there was a fiber line cut between Dallas and LA which is highly affecting Internet traffic.<br>>><br>>> We will continue to work with our datacenter and provide updates. We will receive an estimate when it will be repaired in about 2 hours.<br>>><br>>> We appreciate your patience with the Internet.<br>><br>><br>> --<br>> Anthony Ledesma | FireHost<br>><br>> From: <a href="javascript:;">outages-bounces@outages.org</a> [mailto:<a href="javascript:;">outages-bounces@outages.org</a>] On Behalf Of Martin Hepworth<br>> Sent: Thursday, April 19, 2012 12:43 PM<br>> To: <a href="javascript:;">outages@outages.org</a><br>> Subject: [outages] Rightnow reporting several lines doen in bay area<br>><br>> Anyone got any further info, they are pointing the finger at their providers<br>><br>> --<br>> Martin<br>><br>><br>> --<br>> --<br>> Martin Hepworth<br>> Oxford, UK<br>><br>> _______________________________________________<br>> Outages mailing list<br>> <a href="javascript:;">Outages@outages.org</a><br>> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>><br>> _______________________________________________<br>> Outages mailing list<br>> <a href="javascript:;">Outages@outages.org</a><br>> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p></div><p class=MsoNormal><br><br>-- <br>-- <br>Martin Hepworth<br>Oxford, UK<o:p></o:p></p></div></body></html>