[outages] Level 3 Problems
Joseph Jackson
jjackson at aninetworks.net
Wed Aug 15 11:54:37 EDT 2012
We've seen some packet loss on our customer end points out of NYC over level3 this morning also. Hosted at 60 Hudson.
From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On Behalf Of Dan Stralka
Sent: Wednesday, August 15, 2012 10:15 AM
To: outages at outages.org
Subject: [outages] Level 3 Problems
Seeing high latency on Level 3 out of northern New Jersey. We saw very slow website loads across many sites and a few page timeouts. We also see high latency between our Abovenet circuit and Level 3 where they peer. Level 3 tech support confirms latency problems and did say a lot of customers are calling about this. See traceroutes below:
Level 3 Router:
Type escape sequence to abort.
Tracing the route to france.com<http://france.com> (96.126.117.104)
VRF info: (vrf in name/id, vrf out name/id)
1 159.63.23.213 1 msec 1 msec 1 msec
2 64.215.195.230 [AS 3549] 100 msec 103 msec 114 msec
3 ae-4-90.edge3.NewYork1.Level3.net<http://ae-4-90.edge3.NewYork1.Level3.net> (4.69.155.209) [AS 3356] 49 msec 38 msec
ae-3-80.edge3.NewYork1.Level3.net<http://ae-3-80.edge3.NewYork1.Level3.net> (4.69.155.145) [AS 3356] 35 msec
4 te2-5.bbr01.tl01.nyc01.networklayer.com<http://te2-5.bbr01.tl01.nyc01.networklayer.com> (4.26.34.22) [AS 3356] 99 msec 97 msec 98 msec
5 ae1.bbr01.eq01.chi01.networklayer.com<http://ae1.bbr01.eq01.chi01.networklayer.com> (173.192.18.132) [AS 36351] [MPLS: Label 348745 Exp 0] 119 msec * 119 msec
6 ae20.bbr01.eq01.dal03.networklayer.com<http://ae20.bbr01.eq01.dal03.networklayer.com> (173.192.18.136) [AS 36351] 77 msec 76 msec 128 msec
7 po31.dsr01.dllstx3.networklayer.com<http://po31.dsr01.dllstx3.networklayer.com> (173.192.18.225) [AS 36351] 76 msec 79 msec 80 msec
8 te2-4.dsr01.dllstx2.networklayer.com<http://te2-4.dsr01.dllstx2.networklayer.com> (70.87.255.110) [AS 21844] [MPLS: Label 991 Exp 0] 73 msec 69 msec 72 msec
9 po1.car01.dllstx2.networklayer.com<http://po1.car01.dllstx2.networklayer.com> (70.87.254.74) [AS 21844] 76 msec 76 msec 82 msec
10 router1-dal.linode.com<http://router1-dal.linode.com> (67.18.7.90) [AS 21844] 137 msec 142 msec *
NJASR1#trace 4.69.151.49
Type escape sequence to abort.
Tracing the route to ge-7-15-19.car1.Atlanta1.Level3.net<http://ge-7-15-19.car1.Atlanta1.Level3.net> (4.69.151.49)
VRF info: (vrf in name/id, vrf out name/id)
1 159.63.23.213 1 msec 1 msec 1 msec
2 64.215.195.230 [AS 3549] 94 msec * *
3 vlan60.csw1.NewYork1.Level3.net<http://vlan60.csw1.NewYork1.Level3.net> (4.69.155.62) [AS 3356] 34 msec * 36 msec
4 ae-71-71.ebr1.NewYork1.Level3.net<http://ae-71-71.ebr1.NewYork1.Level3.net> (4.69.134.69) [AS 3356] 30 msec
ae-61-61.ebr1.NewYork1.Level3.net<http://ae-61-61.ebr1.NewYork1.Level3.net> (4.69.134.65) [AS 3356] 100 msec
ae-71-71.ebr1.NewYork1.Level3.net<http://ae-71-71.ebr1.NewYork1.Level3.net> (4.69.134.69) [AS 3356] 34 msec
5 *
ae-10-10.ebr2.Washington12.Level3.net<http://ae-10-10.ebr2.Washington12.Level3.net> (4.69.148.50) [AS 3356] 36 msec 46 msec
6 *
ae-1-100.ebr1.Washington12.Level3.net<http://ae-1-100.ebr1.Washington12.Level3.net> (4.69.143.213) [AS 3356] 122 msec 108 msec
7 ae-6-6.ebr1.Atlanta2.Level3.net<http://ae-6-6.ebr1.Atlanta2.Level3.net> (4.69.148.105) [AS 3356] 132 msec * *
8 ae-11-51.car1.Atlanta1.Level3.net<http://ae-11-51.car1.Atlanta1.Level3.net> (4.69.150.3) [AS 3356] 23 msec * 23 msec
Abovenet (for comparison):
Type escape sequence to abort.
Tracing the route to france.com<http://france.com> (96.126.117.104)
1 209.66.88.145 1 msec 1 msec 1 msec
2 xe-4-1-0.cr1.lga5.us.above.net<http://xe-4-1-0.cr1.lga5.us.above.net> (64.125.31.170) [AS 6461] [MPLS: Label 677655 Exp 0] 1 msec 1 msec 1 msec
3 xe-1-0-1.er3.lga5.us.above.net<http://xe-1-0-1.er3.lga5.us.above.net> (64.125.24.165) [AS 6461] 1 msec 1 msec 1 msec
4 above-ntt-2.lga5.us.above.net<http://above-ntt-2.lga5.us.above.net> (64.125.13.182) [AS 6461] 1 msec 1 msec 1 msec
5 ae-3.r05.nycmny01.us.bb.gin.ntt.net<http://ae-3.r05.nycmny01.us.bb.gin.ntt.net> (129.250.4.204) [AS 2914] 2 msec 1 msec 2 msec
6 xe-1-0-0.bbr01.tl01.nyc01.networklayer.com<http://xe-1-0-0.bbr01.tl01.nyc01.networklayer.com> (165.254.16.2) [AS 2914] 1 msec 1 msec 1 msec
7 ae7.bbr02.tl01.nyc01.networklayer.com<http://ae7.bbr02.tl01.nyc01.networklayer.com> (173.192.18.177) [AS 36351] [MPLS: Label 400085 Exp 0] 1 msec 2 msec 1 msec
8 ae1.bbr01.eq01.chi01.networklayer.com<http://ae1.bbr01.eq01.chi01.networklayer.com> (173.192.18.132) [AS 36351] [MPLS: Label 347833 Exp 0] 20 msec 40 msec 34 msec
9 ae20.bbr01.eq01.dal03.networklayer.com<http://ae20.bbr01.eq01.dal03.networklayer.com> (173.192.18.136) [AS 36351] 44 msec 42 msec 45 msec
10 po31.dsr02.dllstx3.networklayer.com<http://po31.dsr02.dllstx3.networklayer.com> (173.192.18.227) [AS 36351] 42 msec
po31.dsr01.dllstx3.networklayer.com<http://po31.dsr01.dllstx3.networklayer.com> (173.192.18.225) [AS 36351] 40 msec 41 msec
11 te2-4.dsr02.dllstx2.networklayer.com<http://te2-4.dsr02.dllstx2.networklayer.com> (70.87.255.126) [AS 21844] [MPLS: Label 504 Exp 0] 45 msec
te2-4.dsr01.dllstx2.networklayer.com<http://te2-4.dsr01.dllstx2.networklayer.com> (70.87.255.110) [AS 21844] [MPLS: Label 991 Exp 0] 44 msec 40 msec
12 po2.car01.dllstx2.networklayer.com<http://po2.car01.dllstx2.networklayer.com> (70.87.254.78) [AS 21844] 40 msec
po1.car01.dllstx2.networklayer.com<http://po1.car01.dllstx2.networklayer.com> (70.87.254.74) [AS 21844] 44 msec
po2.car01.dllstx2.networklayer.com<http://po2.car01.dllstx2.networklayer.com> (70.87.254.78) [AS 21844] 43 msec
13 router1-dal.linode.com<http://router1-dal.linode.com> (67.18.7.90) [AS 21844] 42 msec 41 msec 44 msec
Tracing the route to ge-7-15-19.car1.Atlanta1.Level3.net<http://ge-7-15-19.car1.Atlanta1.Level3.net> (4.69.151.49)
1 66.6.235.11 2 msec 2 msec 2 msec
2 209.66.88.149 [AS 6461] 2 msec 1 msec 2 msec
3 xe-3-3-0.cr2.lga5.us.above.net<http://xe-3-3-0.cr2.lga5.us.above.net> (64.125.28.218) [AS 6461] [MPLS: Label 769773 Exp 0] 2 msec 2 msec 1 msec
4 xe-2-2-0.cr2.dca2.us.above.net<http://xe-2-2-0.cr2.dca2.us.above.net> (64.125.26.105) [AS 6461] [MPLS: Label 590170 Exp 0] 47 msec 6 msec 6 msec
5 xe-1-0-1.er2.iad10.us.above.net<http://xe-1-0-1.er2.iad10.us.above.net> (64.125.26.242) [AS 6461] 6 msec 7 msec 7 msec
6 above-level3.iad10.us.above.net<http://above-level3.iad10.us.above.net> (64.125.13.234) [AS 6461] 33 msec 32 msec 32 msec
7 vlan60.csw1.Washington1.Level3.net<http://vlan60.csw1.Washington1.Level3.net> (4.69.149.62) [AS 3356] 105 msec
vlan70.csw2.Washington1.Level3.net<http://vlan70.csw2.Washington1.Level3.net> (4.69.149.126) [AS 3356] 45 msec *
8 ae-61-61.ebr1.Washington1.Level3.net<http://ae-61-61.ebr1.Washington1.Level3.net> (4.69.134.129) [AS 3356] 31 msec
ae-81-81.ebr1.Washington1.Level3.net<http://ae-81-81.ebr1.Washington1.Level3.net> (4.69.134.137) [AS 3356] 20 msec
ae-71-71.ebr1.Washington1.Level3.net<http://ae-71-71.ebr1.Washington1.Level3.net> (4.69.134.133) [AS 3356] 26 msec
9 ae-2-2.ebr3.Atlanta2.Level3.net<http://ae-2-2.ebr3.Atlanta2.Level3.net> (4.69.132.85) [AS 3356] 51 msec 51 msec 50 msec
10 4.69.148.242 [AS 3356] 125 msec
ae-73-73.ebr2.Atlanta2.Level3.net<http://ae-73-73.ebr2.Atlanta2.Level3.net> (4.69.148.254) [AS 3356] 53 msec 48 msec
11 ae-11-51.car1.Atlanta1.Level3.net<http://ae-11-51.car1.Atlanta1.Level3.net> (4.69.150.3) [AS 3356] 61 msec * 44 msec
Looking glass trace from Level 3:
1 ge-6-17-17.car1.Atlanta1.Level3.net<http://ge-6-17-17.car1.Atlanta1.Level3.net> (4.69.151.45) 12 msec
ge-6-13-18.car2.Atlanta1.Level3.net<http://ge-6-13-18.car2.Atlanta1.Level3.net> (4.69.151.73) 16 msec
ge-6-17-17.car1.Atlanta1.Level3.net<http://ge-6-17-17.car1.Atlanta1.Level3.net> (4.69.151.45) 12 msec
2 ae-2-52.edge5.Atlanta2.Level3.net<http://ae-2-52.edge5.Atlanta2.Level3.net> (4.69.150.78) 28 msec * *
3 64.208.27.209 [AS3549 {GBLX}] 12 msec 12 msec 16 msec
4 67.16.131.134 [AS3549 {GBLX}] 60 msec 64 msec
pos3-0-0-2488M.asr1.JFK1.gblx.net<http://pos3-0-0-2488M.asr1.JFK1.gblx.net> (67.16.131.138) [AS3549 {GBLX}] 128 msec
5 159.63.23.214 [AS3549 {GBLX}] 64 msec 60 msec 128 msec
6 * *
66.6.224.47 [AS40073 {NBA-ASN-USA}] !filtered
-- Webmail
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20120815/6cc1824d/attachment.htm>
More information about the Outages
mailing list