[outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015

Fri Nov 20 11:11:02 EST 2015

You are not alone.

                             My traceroute  [v0.85]
nocstation-13 (0.0.0.0)                                Fri Nov 20 10:08:57
2015
Ping Bit Pattern: ^[[B^[[B
Pattern Range: 0(0x00)-255(0xff), <0 random.
 *Host                                Loss%   Snt   Last   Avg  Best  Wrst
StDev*
 1. xxxxxxxxxxxxxxxxxxxxxx       0.0%   200    0.6   0.3   0.2   0.6   0.0
 2. xxxxxxxxxxxxxxxxxxxx            0.0%   199    1.3   1.7   1.2   2.4
0.0
 3. open.airstreamcomm.net            0.0%   199    3.6   5.2   3.3  20.3
3.7
 4. chi-b21-link.telia.net            0.0%   199   10.6  12.0  10.2  25.7
3.4
 5. level3-ic-306666-chi-b21.c.telia 27.3%   199   10.9 2011.  10.5 21109
4930.
 *6. ae-8-3603.ebr4.Chicago2.Level3.n 89.9%   199   12.8  14.7  12.8  28.9
  4.1*
 7. ae-14-14.ebr1.Dallas1.Level3.net 17.1%   199   87.7  89.2  87.4 103.9
3.7
 8. ae-91-91.csw4.Dallas1.Level3.net 15.7%   199  100.4  88.6  87.1 102.8
2.8
 9. ae-93-93.ebr3.Dallas1.Level3.net 18.6%   199   87.8  89.0  87.3 103.6
3.4
10. ae-3-3.ebr2.LosAngeles1.Level3.n 21.1%   199   88.1  89.5  87.7 104.4
3.7
11. ae-82-82.csw3.LosAngeles1.Level3 20.6%   199   87.7  89.2  87.3 103.8
3.8
12. ae-3-80.edge6.LosAngeles1.Level3 17.6%   199   92.1  88.4  87.2 102.9
2.6
13. WINKY-NETWO.edge6.LosAngeles1.Le 18.6%   199   87.8  92.5  87.5 213.7
 16.4
14. 38.102.236.67                    17.1%   199  101.2  91.2  89.0 110.1
3.8

Here's everything from that ticket yesterday with something relevant for
today:

Reason for Outage (RFO) Summary: The Transport NOC detected a loss of
signal between Odell, IL and Elwood, IL impacting unprotected services.
Field Services was dispatched to both the Odell, IL and Elwood, IL
locations to continue efforts to isolate the suspected fiber cut impacting
unprotected services. A splice crew was already been engaged to be
available for repairs. Field Services arrived onsite in Elwood, IL and
continued with investigation of the suspected fiber cut. Investigations
from the Elwood, IL site indicate the fiber cut was closer to the Odell, IL
side of the path. There was known railroad construction in that area that
was suspected to be the cause of fiber damage. Field Services arrived
onsite in Odell, IL. Investigations indicated the fault was approximately
1.4 miles from the site. There was a construction crew performing work
along the railroad track that damaged a Level 3 fiber cable. Field Services
began working to pinpoint the damage point within the construction area for
further evaluation. Field Services began working to access hand holes in
order to continue investigation and restoration efforts. Field Services are
began excavating to the hand hole with a backhoe. Field Services accessed
one hand hole and began working on accessing a second. Field Services
excavated three hand holes to reach the 108-count fiber cable for repair.
Field Services successfully pulled the slack through the handholes at both
ends and exposed the fibers. Field Services advised that splicing commenced
at both ends of the fiber. The Transport NOC observed all alarms clear. *The
Transport NOC advised that some alarms remain but have determined that the
issue was unrelated. Field Services have advised that splicing has been
completed. The Technical Service Center confirmed service restoral with the
customers. *If additional issues are experienced, please contact the Level
3 Technical Service Center at 1-877-4LEVEL3.

I'll create a new thread if I get a new master ticket.

On Fri, Nov 20, 2015 at 10:01 AM, Erik Sundberg <ESundberg at nitelusa.com>
wrote:

> We are still seeing some issue from yesterday’s fiber cut on Level3. Some
> circuit’s never came back up and other went back down this morning. I am
> working with Level3 on this now.
>
>
>
> Wondering if anyone else is still having issues between Chicago and St
> Louis?
>
>
>
>
>
>
>
> *From:* Outages [mailto:outages-bounces at outages.org] *On Behalf Of *Thom
> Cleary via Outages
> *Sent:* Thursday, November 19, 2015 7:01 PM
> *To:* frnkblk at iname.com; outages at outages.org
> *Subject:* [outages] XO / Level 3 Outage | Chicago - St Louis | 11/19/2015
>
>
>
> Here's the latest:
>
>
> *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 -
> Event
> Field Services continue preparation efforts at this time.
>
> 11/20/2015 00:40:36 GMT -
> *** CASCADED EXTERNAL NOTES 20-Nov-2015 00:36:21 GMT From CASE: 10003567 -
> Event
> Field Services continue preparation efforts at this time.
>
>
> 11/19/2015 23:47:36 GMT -
> *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:43:49 GMT From CASE: 10003567 -
> Event
> Restoration efforts are still ongoing, an estimated restoral time has not
> been provided.
>
>
> 11/19/2015 23:05:21 GMT -
> *** CASCADED EXTERNAL NOTES 19-Nov-2015 23:00:53 GMT From CASE: 10003567 -
> Event
> Field Services have excavated three hand holes to reach the 108-count
> fiber cable for repair. Efforts are ongoing at the site with no estimated
> restoral time currently available.
>
>
> 11/19/2015 22:29:51 GMT -
> *** CASCADED EXTERNAL NOTES 19-Nov-2015 22:25:57 GMT From CASE: 10003567 -
> Event
> Field Services have accessed one hand hole and are working on accessing a
> second. Evaluation of the cable damage is underway.
>
>
> 11/19/2015 21:51:05 GMT -
> *** CASCADED EXTERNAL NOTES 19-Nov-2015 21:47:47 GMT From CASE: 10003567 -
> Event
> Field Services are now excavating to the hand hole with a backhoe. Once
> access is available, evaluation of the fault will continue.
>
>
> 11/19/2015 20:54:23 GMT -
> *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:50:55 GMT From CASE: 10003567 -
> Event
> Field Services are manually digging to access the hand hole closest to the
> suspected damage point. Once accessed, Field Services will be able to
> better evaluate the issue.
>
>
> 11/19/2015 20:12:15 GMT -
> *** CASCADED EXTERNAL NOTES 19-Nov-2015 20:08:11 GMT From CASE: 10003567 -
> Event
> The plow work at the site has made isolation of the damage point difficult
> to pinpoint. Field Services are working to access hand holes in order to
> continue investigation and restoration efforts.
>
>
> 11/19/2015 19:08:37 GMT -
> *** CASCADED EXTERNAL NOTES 19-Nov-2015 19:04:13 GMT From CASE: 10003567 -
> Event
> Field Services are onsite in the construction area working to locate the
> exact damage point. Replacement cable is being dispatched to the site to
> have available for repair and should arrive by 20:00 GMT.
>
>
>
>
>
>
>
> ------------------------------
>
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
> or previous e-mail messages attached to it may contain confidential
> information that is legally privileged. If you are not the intended
> recipient, or a person responsible for delivering it to the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of any of the information contained in or attached to
> this transmission is STRICTLY PROHIBITED. If you have received this
> transmission in error please notify the sender immediately by replying to
> this e-mail. You must destroy the original transmission and its attachments
> without reading or saving in any manner. Thank you.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20151120/7317cef5/attachment.htm>