[outages] Level 3 Problems

Wed Aug 15 11:28:31 EDT 2012

We have TOTAL failure on our level 3 voice ckts. T1's down since 0900

From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
Behalf Of Dan Stralka
Sent: Wednesday, August 15, 2012 11:15 AM
To: outages at outages.org
Subject: [outages] Level 3 Problems

Seeing high latency on Level 3 out of northern New Jersey.  We saw very slow
website loads across many sites and a few page timeouts.  We also see high
latency between our Abovenet circuit and Level 3 where they peer. Level 3
tech support confirms latency problems and did say a lot of customers are
calling about this. See traceroutes below:

Level 3 Router:

Type escape sequence to abort.

Tracing the route to france.com (96.126.117.104)

VRF info: (vrf in name/id, vrf out name/id)

  1 159.63.23.213 1 msec 1 msec 1 msec

  2 64.215.195.230 [AS 3549] 100 msec 103 msec 114 msec

  3 ae-4-90.edge3.NewYork1.Level3.net (4.69.155.209) [AS 3356] 49 msec 38
msec

    ae-3-80.edge3.NewYork1.Level3.net (4.69.155.145) [AS 3356] 35 msec

  4 te2-5.bbr01.tl01.nyc01.networklayer.com (4.26.34.22) [AS 3356] 99 msec
97 msec 98 msec

  5 ae1.bbr01.eq01.chi01.networklayer.com (173.192.18.132) [AS 36351] [MPLS:
Label 348745 Exp 0] 119 msec *  119 msec

  6 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) [AS 36351] 77
msec 76 msec 128 msec

  7 po31.dsr01.dllstx3.networklayer.com (173.192.18.225) [AS 36351] 76 msec
79 msec 80 msec

  8 te2-4.dsr01.dllstx2.networklayer.com (70.87.255.110) [AS 21844] [MPLS:
Label 991 Exp 0] 73 msec 69 msec 72 msec

  9 po1.car01.dllstx2.networklayer.com (70.87.254.74) [AS 21844] 76 msec 76
msec 82 msec

 10 router1-dal.linode.com (67.18.7.90) [AS 21844] 137 msec 142 msec *

NJASR1#trace 4.69.151.49

Type escape sequence to abort.

Tracing the route to ge-7-15-19.car1.Atlanta1.Level3.net (4.69.151.49)

VRF info: (vrf in name/id, vrf out name/id)

  1 159.63.23.213 1 msec 1 msec 1 msec

  2 64.215.195.230 [AS 3549] 94 msec *  *

  3 vlan60.csw1.NewYork1.Level3.net (4.69.155.62) [AS 3356] 34 msec *  36
msec

  4 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) [AS 3356] 30 msec

    ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65) [AS 3356] 100 msec

    ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) [AS 3356] 34 msec

  5  *

    ae-10-10.ebr2.Washington12.Level3.net (4.69.148.50) [AS 3356] 36 msec 46
msec

  6  *

    ae-1-100.ebr1.Washington12.Level3.net (4.69.143.213) [AS 3356] 122 msec
108 msec

  7 ae-6-6.ebr1.Atlanta2.Level3.net (4.69.148.105) [AS 3356] 132 msec *  *

  8 ae-11-51.car1.Atlanta1.Level3.net (4.69.150.3) [AS 3356] 23 msec *  23
msec

Abovenet (for comparison):

Type escape sequence to abort.

Tracing the route to france.com (96.126.117.104)

  1 209.66.88.145 1 msec 1 msec 1 msec

  2 xe-4-1-0.cr1.lga5.us.above.net (64.125.31.170) [AS 6461] [MPLS: Label
677655 Exp 0] 1 msec 1 msec 1 msec

  3 xe-1-0-1.er3.lga5.us.above.net (64.125.24.165) [AS 6461] 1 msec 1 msec 1
msec

  4 above-ntt-2.lga5.us.above.net (64.125.13.182) [AS 6461] 1 msec 1 msec 1
msec

  5 ae-3.r05.nycmny01.us.bb.gin.ntt.net (129.250.4.204) [AS 2914] 2 msec 1
msec 2 msec

  6 xe-1-0-0.bbr01.tl01.nyc01.networklayer.com (165.254.16.2) [AS 2914] 1
msec 1 msec 1 msec

  7 ae7.bbr02.tl01.nyc01.networklayer.com (173.192.18.177) [AS 36351] [MPLS:
Label 400085 Exp 0] 1 msec 2 msec 1 msec

  8 ae1.bbr01.eq01.chi01.networklayer.com (173.192.18.132) [AS 36351] [MPLS:
Label 347833 Exp 0] 20 msec 40 msec 34 msec

  9 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) [AS 36351] 44
msec 42 msec 45 msec

 10 po31.dsr02.dllstx3.networklayer.com (173.192.18.227) [AS 36351] 42 msec

    po31.dsr01.dllstx3.networklayer.com (173.192.18.225) [AS 36351] 40 msec
41 msec

 11 te2-4.dsr02.dllstx2.networklayer.com (70.87.255.126) [AS 21844] [MPLS:
Label 504 Exp 0] 45 msec

    te2-4.dsr01.dllstx2.networklayer.com (70.87.255.110) [AS 21844] [MPLS:
Label 991 Exp 0] 44 msec 40 msec

 12 po2.car01.dllstx2.networklayer.com (70.87.254.78) [AS 21844] 40 msec

    po1.car01.dllstx2.networklayer.com (70.87.254.74) [AS 21844] 44 msec

    po2.car01.dllstx2.networklayer.com (70.87.254.78) [AS 21844] 43 msec

 13 router1-dal.linode.com (67.18.7.90) [AS 21844] 42 msec 41 msec 44 msec

Tracing the route to ge-7-15-19.car1.Atlanta1.Level3.net (4.69.151.49)

  1 66.6.235.11 2 msec 2 msec 2 msec

  2 209.66.88.149 [AS 6461] 2 msec 1 msec 2 msec

  3 xe-3-3-0.cr2.lga5.us.above.net (64.125.28.218) [AS 6461] [MPLS: Label
769773 Exp 0] 2 msec 2 msec 1 msec

  4 xe-2-2-0.cr2.dca2.us.above.net (64.125.26.105) [AS 6461] [MPLS: Label
590170 Exp 0] 47 msec 6 msec 6 msec

  5 xe-1-0-1.er2.iad10.us.above.net (64.125.26.242) [AS 6461] 6 msec 7 msec
7 msec

  6 above-level3.iad10.us.above.net (64.125.13.234) [AS 6461] 33 msec 32
msec 32 msec

  7 vlan60.csw1.Washington1.Level3.net (4.69.149.62) [AS 3356] 105 msec

    vlan70.csw2.Washington1.Level3.net (4.69.149.126) [AS 3356] 45 msec *

  8 ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) [AS 3356] 31 msec

    ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) [AS 3356] 20 msec

    ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) [AS 3356] 26 msec

  9 ae-2-2.ebr3.Atlanta2.Level3.net (4.69.132.85) [AS 3356] 51 msec 51 msec
50 msec

 10 4.69.148.242 [AS 3356] 125 msec

    ae-73-73.ebr2.Atlanta2.Level3.net (4.69.148.254) [AS 3356] 53 msec 48
msec

 11 ae-11-51.car1.Atlanta1.Level3.net (4.69.150.3) [AS 3356] 61 msec *  44
msec

Looking glass trace from Level 3:

  1 ge-6-17-17.car1.Atlanta1.Level3.net (4.69.151.45) 12 msec
    ge-6-13-18.car2.Atlanta1.Level3.net (4.69.151.73) 16 msec
    ge-6-17-17.car1.Atlanta1.Level3.net (4.69.151.45) 12 msec
  2 ae-2-52.edge5.Atlanta2.Level3.net (4.69.150.78) 28 msec *  * 
  3 64.208.27.209 [AS3549 {GBLX}] 12 msec 12 msec 16 msec
  4 67.16.131.134 [AS3549 {GBLX}] 60 msec 64 msec
    pos3-0-0-2488M.asr1.JFK1.gblx.net (67.16.131.138) [AS3549 {GBLX}] 128
msec
  5 159.63.23.214 [AS3549 {GBLX}] 64 msec 60 msec 128 msec
  6  *  * 
    66.6.224.47 [AS40073 {NBA-ASN-USA}] !filtered 

-- Webmail

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20120815/08a8b0c9/attachment.htm>