[outages] Level3 Looking Glassdown

Tue Jan 31 13:15:42 EST 2017

There is a Level 3 cut in Houston, TX, of a 168-count and a 96-count cable, 
that is apparently impacting lots of east-west traffic.  MN seems a bit far 
from Houston, TX, to be impacted, but perhaps?

*** CASCADED EXTERNAL NOTES 31-Jan-2017 17:48:35 GMT From CASE: 12152675 - Event
The Transport NOC has advised that Field Services have confirmed that both the 168 count cable and the 96 count cable onsite have been
cut.  Outside plant engineers have pulled 80 feet of slack on the 168 count cable from the existing splice point, which is being used
to repair that fiber.  At this time, the fiber is being prepared for the ensuing fiber repair.  Outside plant engineers were unable to
pull any slack on the 96 count cable from the splice point.  As such, the splice crew is preparing to splice a temporary cable throw
from the break point to the existing splice point.  There is no ETTR for this case at this time.

01/31/2017 16:21:26 GMT -
*** CASCADED EXTERNAL NOTES 31-Jan-2017 16:19:28 GMT From CASE: 12152675 - Event
The Transport NOC has advised that the damaged cable is a 168 count cable that was struck by a backhoe during a department of
transportation construction project onsite.  The construction project may have also damaged a 96 count cable onsite.  Field Services
and the Transport NOC are working to confirm the damage to the second cable.   The excavation/splicing contractor is still en route;
however outside plant engineers are still unable to provide an ETA for the contractors.  Outside plant engineers have begun excavating
effort onsite with the available construction equipment and have identified an existing splice point nearby. At this time, effort to
pull the necessary slack from the existing splice point are underway.

01/31/2017 15:04:17 GMT - 01/31/2017 15:03:31 GMT - The Transport NOC has advised that Field Services have located the fiber damage
and determined the break was the result of a construction project.  Outside plant engineers are onsite and are waiting for the
splicing contractors to arrive.  There is no ETA for the splicing contractors at this time.

01/31/2017 14:03:16 GMT - The Transport NOC has advised that Field Services have arrived onsite and are preparing to test the affected
fiber to confirm the existence and location of the damaged cable. Fiber testing will commence momentarily.

01/31/2017 13:59:00 GMT - A fiber cut in Houston, TX is impacting unprotected services. The Transport NOC has engaged Field Services
to dispatch a field technician to the failure site.  There is no ETA for the field technician at this time. Please be advised that
updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated
to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident
review, details may evolve.

-- 
Hugo

On Tue 2017-Jan-31 18:07:29 +0000, Derek J. Nicholls via Outages <outages at outages.org> wrote:

>We had to turn down our BGP with Level3 due to almost complete loss for ~5 minutes.
>I also have a layer2 circuit with them which has come back up since but is experiencing a lot of loss.
>We’re out of Eagan, MN
>
>Derek Nicholls
>Sr. Network Administrator | Tempworks Software & Payroll Funding
>Office: 651-452-0366
>3140 Neil Armstrong Blvd Suite 205 | Eagan | Minnesota | 55121
>dnicholls at tempworks.com<mailto:dnicholls at tempworks.com> | www.tempworks.com<http://www.tempworks.com/>
>
>From: Outages [mailto:outages-bounces at outages.org] On Behalf Of Ca By via Outages
>Sent: Tuesday, January 31, 2017 12:01 PM
>To: outages at outages.org
>Subject: [outages] Level3 Looking Glassdown
>
>Level3 folks,  please fix.
>
> wget -T 5 lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,65ZeY0snVET9qaPqRdvh2Sa3RT7MZE61MghnAkIys49KsaIlZSkH-GtXO3UbHtv2Tz-iL-gSz4v2aleWIgoksfVGKFUGXKhWXDqG_i1xqYPn43LsI-D1ZA,,&typo=1>
>converted 'http://lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,4pEiSraAcLmXw830KB1JR7WeRwgOlgQFIBIPE8Gm84BKrZIFHpG7QPLdTZ4G_AdEqSy2NiglSzfexPkGh0EWnfS7KZiGVjS4ciGLmXz64GQRdupa7kAV&typo=1>' (ANSI_X3.4-1968) -> 'http://lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,zSJO5Pjj73AErWit4kheKsWg3pyJroiNZICzn480acGF2WsydN64mQ54LYkLwJ0wqWsoX2n11UU0gBoyCU5Hpcw3MWxQGnIOxrjnbU2CTxOt4HAebWsL86dX&typo=1>' (UTF-8)
>--2017-01-31 12:58:57--  http://lg.level3.net/<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net/&c=E,1,LE6Ov7vn2fjHkJN2Q1Iu3UvrqWFeoCcL2-OHR34YathQl0arHzdDrK7AhWKZTjSqvd6YRtz11eU7J3EFdosafQqRtHMt-y_stnSujqGPhpRkcQ,,&typo=1>
>Resolving lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,RQ3mPbIz98skzt2ykO8BCSVmUwynT2-OPtvgxIcq-Rakc53Vp83Yj7VYZ-kBD_TVF1NcVuIeFPlTW5NwmZKVhnjpH2j6QPNvhkDpD_Km0du0fdF3DE8iBjLFXA,,&typo=1> (lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,63yd2QpqHgbEbSui2UG9EXJP7kDPyUjxo5FxuBFeCdxxIGKRmeNjzDw3xCrurqHne6N9nb9tSBJTijm74oLNTb4inp3D_Pcf-inup6CxE2-J10dODroe&typo=1>)... 209.245.18.181
>Connecting to lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,CvU5bdnbR5iloFh9gq6ytXW2dwqgB7rZcr5lkZemNddocPBuOA176nIIq7aVo1eUxUiOnZyEdzIIMJMl6LRZxlliVpqBTbAXVvS0TiyCpuBQbuw,&typo=1> (lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,1mCfvG2agTr2J86kkpqv1SWVR8weolRUs0iD4SUpZGsLlS9a3kgRi2_LnCv8MTYjOayj9rqxlNP_Tnuf8riIc6tMWn1rCGoFz-UtyAxYgJsVgty4&typo=1>)|209.245.18.181|:80... failed: Connection timed out.
>Retrying.
>
>
>traceroute lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,ygQIZyLx3YeBLoyeSml5g1LLLnbzrdFkSYw1IPJ7MYAw3xnbCPJu5o4_mq3j72d-xteZistsl4CX9oQRs8o9vIPefDFo1znO962KiJGxb2RDTg,,&typo=1>
>traceroute to lg.level3.net<https://linkprotect.cudasvc.com/url?a=http://lg.level3.net&c=E,1,lvNtrJjWmsdmwMjC26klIHSgkEgQNn8c684jZ787A4WUsvJlWB113O2VAzhXPjOu2QzyXkphGPlCGd-YZ4Y--7T4SIOWBfXbwJdMsESd3A,,&typo=1> (209.245.18.181), 30 hops max, 60 byte packets
> 1  * * *
> 2  xe-0-1-0-17.r04.atlnga05.us.bb.gin.ntt.net<https://linkprotect.cudasvc.com/url?a=http://xe-0-1-0-17.r04.atlnga05.us.bb.gin.ntt.net&c=E,1,u2dYqQiQcFjAQ0JYGBEzE3-MtU-PYnHvK5jbtKaLII9Qy56WQwaGJtvIloTA00Mxu_2AjwoYUfDOhq2Re_q5dK7YmwjyLnvF9IjBwOA8vwLUxYySrsk,&typo=1> (129.250.203.221)  1.284 ms  1.277 ms ae7-300.cr2-atl1.ip4.gtt.net<https://linkprotect.cudasvc.com/url?a=http://ae7-300.cr2-atl1.ip4.gtt.net&c=E,1,YpIlJ_1gNF4En7uHt16oCzOdtuETGxfx_YY4isLvhloZU2LC3KWTD-y4F-1Il_SsQ3RTsqvk78ffUOMVwBwS0ZsqzJ-O85JRPR6jmLHd7_5_e_1-&typo=1> (173.205.55.97)  0.241 ms
> 3  ae-4.r03.atlnga05.us.bb.gin.ntt.net<https://linkprotect.cudasvc.com/url?a=http://ae-4.r03.atlnga05.us.bb.gin.ntt.net&c=E,1,CKY1IS6w12E6GEvwL3JejiYzxBuA55OSlIEb0tnDaxSVeg9mhZxU-wf6L__WuKmKxhWBn47D-SyiZaXImzXyXrVrgC1Ed4-R_LZsDBtt0nRUS5E,&typo=1> (129.250.5.204)  0.427 ms ae4.edge2.Atlanta4.Level3.net<https://linkprotect.cudasvc.com/url?a=http://ae4.edge2.Atlanta4.Level3.net&c=E,1,fDraKWbUYry75w12FGbrDKixsWxRFYqrTAw-kbx2nERVfPcPf2q1ebrUxU4kqZGjYK6UUVHzv_sm8UZJnR08PPi6ct1hcUxdsInWhB-kmRPJaupxn3WxKwEpVA,,&typo=1> (4.68.63.157)  0.222 ms ae-4.r03.atlnga05.us.bb.gin.ntt.net<https://linkprotect.cudasvc.com/url?a=http://ae-4.r03.atlnga05.us.bb.gin.ntt.net&c=E,1,hAWB87_8a3WaKyBRueB6UrEfpNZbjJvYuEIDB7mf5zhJoAWy0SuQlwi3lCdN5JHR7Py3TFbA7JXK7e_a5Mt1BWr_R7qfv10DJjGRskcJHA6ZqPbUddFp&typo=1> (129.250.5.204)  0.464 ms
> 4  * * ae-1-51.edge6.Denver1.Level3.net<https://linkprotect.cudasvc.com/url?a=http://ae-1-51.edge6.Denver1.Level3.net&c=E,1,yElQnfHCp3eYR020r23JE_ZND0SzQWPyWG00Prgu5Oc80Eb68nmqO9M5rvwuHqPBOLpWkE3a5Tc-PMm_9mpIBGJWt8GheXUmIcIcuw0JmSaTKZfvOCUq&typo=1> (4.69.147.74)  34.727 ms
> 5  ae-1-51.edge6.Denver1.Level3.net<https://linkprotect.cudasvc.com/url?a=http://ae-1-51.edge6.Denver1.Level3.net&c=E,1,ERQdEewwLju-Mx1WPVGTYsFWy43HWzvPXw1xyPDWHhMcDsp2Oblj7CSqw-DD0F-KxgZc8GxH_bY753XaximlM97NU_Jy7_W_p0ZHeT5yNQ,,&typo=1> (4.69.147.74)  34.655 ms  34.639 ms  34.641 ms
> 6  * * *
> 7  * * *
> 8  * * *
> 9  * * *
>10  * * *
>
>THIS MESSAGE IS ONLY INTENDED FOR THE USE OF THE INTENDED RECIPIENT(S) AND MAY CONTAIN INFORMATION THAT IS PRIVILEGED, PROPRIETARY AND/OR CONFIDENTIAL. If you are not the intended recipient, you are hereby notified that any review, re-transmission, dissemination, distribution, copying, conversion to hard copy or other use of this communication is strictly prohibited. If you are not the intended recipient and have received this message in error, please notify me by return e-mail and delete this message from your system. TempWorks Software, Inc.

>_______________________________________________
>Outages mailing list
>Outages at outages.org
>https://puck.nether.net/mailman/listinfo/outages

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20170131/91538220/attachment.sig>