Seeing high latency on Level 3 out of northern New Jersey. We saw very slow website loads across many sites and a few page timeouts. We also see high latency between our Abovenet circuit and Level 3 where they peer. Level 3 tech support confirms latency problems and did say a lot of customers are calling about this. See traceroutes below:<div>
<br></div><div>Level 3 Router:</div><div><br></div><div><div>Type escape sequence to abort.</div><div>Tracing the route to <a href="http://france.com">france.com</a> (96.126.117.104)</div><div>VRF info: (vrf in name/id, vrf out name/id)</div>
<div> 1 159.63.23.213 1 msec 1 msec 1 msec</div><div> 2 64.215.195.230 [AS 3549] 100 msec 103 msec 114 msec</div><div> 3 <a href="http://ae-4-90.edge3.NewYork1.Level3.net">ae-4-90.edge3.NewYork1.Level3.net</a> (4.69.155.209) [AS 3356] 49 msec 38 msec</div>
<div> <a href="http://ae-3-80.edge3.NewYork1.Level3.net">ae-3-80.edge3.NewYork1.Level3.net</a> (4.69.155.145) [AS 3356] 35 msec</div><div> 4 <a href="http://te2-5.bbr01.tl01.nyc01.networklayer.com">te2-5.bbr01.tl01.nyc01.networklayer.com</a> (4.26.34.22) [AS 3356] 99 msec 97 msec 98 msec</div>
<div> 5 <a href="http://ae1.bbr01.eq01.chi01.networklayer.com">ae1.bbr01.eq01.chi01.networklayer.com</a> (173.192.18.132) [AS 36351] [MPLS: Label 348745 Exp 0] 119 msec * 119 msec</div><div> 6 <a href="http://ae20.bbr01.eq01.dal03.networklayer.com">ae20.bbr01.eq01.dal03.networklayer.com</a> (173.192.18.136) [AS 36351] 77 msec 76 msec 128 msec</div>
<div> 7 <a href="http://po31.dsr01.dllstx3.networklayer.com">po31.dsr01.dllstx3.networklayer.com</a> (173.192.18.225) [AS 36351] 76 msec 79 msec 80 msec</div><div> 8 <a href="http://te2-4.dsr01.dllstx2.networklayer.com">te2-4.dsr01.dllstx2.networklayer.com</a> (70.87.255.110) [AS 21844] [MPLS: Label 991 Exp 0] 73 msec 69 msec 72 msec</div>
<div> 9 <a href="http://po1.car01.dllstx2.networklayer.com">po1.car01.dllstx2.networklayer.com</a> (70.87.254.74) [AS 21844] 76 msec 76 msec 82 msec</div><div> 10 <a href="http://router1-dal.linode.com">router1-dal.linode.com</a> (67.18.7.90) [AS 21844] 137 msec 142 msec *</div>
<div><br></div><div><div><br></div><div><div>NJASR1#trace 4.69.151.49</div><div>Type escape sequence to abort.</div><div>Tracing the route to <a href="http://ge-7-15-19.car1.Atlanta1.Level3.net">ge-7-15-19.car1.Atlanta1.Level3.net</a> (4.69.151.49)</div>
<div>VRF info: (vrf in name/id, vrf out name/id)</div><div> 1 159.63.23.213 1 msec 1 msec 1 msec</div><div> 2 64.215.195.230 [AS 3549] 94 msec * *</div><div> 3 <a href="http://vlan60.csw1.NewYork1.Level3.net">vlan60.csw1.NewYork1.Level3.net</a> (4.69.155.62) [AS 3356] 34 msec * 36 msec</div>
<div> 4 <a href="http://ae-71-71.ebr1.NewYork1.Level3.net">ae-71-71.ebr1.NewYork1.Level3.net</a> (4.69.134.69) [AS 3356] 30 msec</div><div> <a href="http://ae-61-61.ebr1.NewYork1.Level3.net">ae-61-61.ebr1.NewYork1.Level3.net</a> (4.69.134.65) [AS 3356] 100 msec</div>
<div> <a href="http://ae-71-71.ebr1.NewYork1.Level3.net">ae-71-71.ebr1.NewYork1.Level3.net</a> (4.69.134.69) [AS 3356] 34 msec</div><div> 5 *</div><div> <a href="http://ae-10-10.ebr2.Washington12.Level3.net">ae-10-10.ebr2.Washington12.Level3.net</a> (4.69.148.50) [AS 3356] 36 msec 46 msec</div>
<div> 6 *</div><div> <a href="http://ae-1-100.ebr1.Washington12.Level3.net">ae-1-100.ebr1.Washington12.Level3.net</a> (4.69.143.213) [AS 3356] 122 msec 108 msec</div><div> 7 <a href="http://ae-6-6.ebr1.Atlanta2.Level3.net">ae-6-6.ebr1.Atlanta2.Level3.net</a> (4.69.148.105) [AS 3356] 132 msec * *</div>
<div> 8 <a href="http://ae-11-51.car1.Atlanta1.Level3.net">ae-11-51.car1.Atlanta1.Level3.net</a> (4.69.150.3) [AS 3356] 23 msec * 23 msec</div></div></div><div><br></div><div><br></div><div><br></div><div><br></div><div>
Abovenet (for comparison):</div><div><br></div><div><br></div><div><div>Type escape sequence to abort.</div><div>Tracing the route to <a href="http://france.com">france.com</a> (96.126.117.104)</div><div><br></div><div> 1 209.66.88.145 1 msec 1 msec 1 msec</div>
<div> 2 <a href="http://xe-4-1-0.cr1.lga5.us.above.net">xe-4-1-0.cr1.lga5.us.above.net</a> (64.125.31.170) [AS 6461] [MPLS: Label 677655 Exp 0] 1 msec 1 msec 1 msec</div><div> 3 <a href="http://xe-1-0-1.er3.lga5.us.above.net">xe-1-0-1.er3.lga5.us.above.net</a> (64.125.24.165) [AS 6461] 1 msec 1 msec 1 msec</div>
<div> 4 <a href="http://above-ntt-2.lga5.us.above.net">above-ntt-2.lga5.us.above.net</a> (64.125.13.182) [AS 6461] 1 msec 1 msec 1 msec</div><div> 5 <a href="http://ae-3.r05.nycmny01.us.bb.gin.ntt.net">ae-3.r05.nycmny01.us.bb.gin.ntt.net</a> (129.250.4.204) [AS 2914] 2 msec 1 msec 2 msec</div>
<div> 6 <a href="http://xe-1-0-0.bbr01.tl01.nyc01.networklayer.com">xe-1-0-0.bbr01.tl01.nyc01.networklayer.com</a> (165.254.16.2) [AS 2914] 1 msec 1 msec 1 msec</div><div> 7 <a href="http://ae7.bbr02.tl01.nyc01.networklayer.com">ae7.bbr02.tl01.nyc01.networklayer.com</a> (173.192.18.177) [AS 36351] [MPLS: Label 400085 Exp 0] 1 msec 2 msec 1 msec</div>
<div> 8 <a href="http://ae1.bbr01.eq01.chi01.networklayer.com">ae1.bbr01.eq01.chi01.networklayer.com</a> (173.192.18.132) [AS 36351] [MPLS: Label 347833 Exp 0] 20 msec 40 msec 34 msec</div><div> 9 <a href="http://ae20.bbr01.eq01.dal03.networklayer.com">ae20.bbr01.eq01.dal03.networklayer.com</a> (173.192.18.136) [AS 36351] 44 msec 42 msec 45 msec</div>
<div> 10 <a href="http://po31.dsr02.dllstx3.networklayer.com">po31.dsr02.dllstx3.networklayer.com</a> (173.192.18.227) [AS 36351] 42 msec</div><div> <a href="http://po31.dsr01.dllstx3.networklayer.com">po31.dsr01.dllstx3.networklayer.com</a> (173.192.18.225) [AS 36351] 40 msec 41 msec</div>
<div> 11 <a href="http://te2-4.dsr02.dllstx2.networklayer.com">te2-4.dsr02.dllstx2.networklayer.com</a> (70.87.255.126) [AS 21844] [MPLS: Label 504 Exp 0] 45 msec</div><div> <a href="http://te2-4.dsr01.dllstx2.networklayer.com">te2-4.dsr01.dllstx2.networklayer.com</a> (70.87.255.110) [AS 21844] [MPLS: Label 991 Exp 0] 44 msec 40 msec</div>
<div> 12 <a href="http://po2.car01.dllstx2.networklayer.com">po2.car01.dllstx2.networklayer.com</a> (70.87.254.78) [AS 21844] 40 msec</div><div> <a href="http://po1.car01.dllstx2.networklayer.com">po1.car01.dllstx2.networklayer.com</a> (70.87.254.74) [AS 21844] 44 msec</div>
<div> <a href="http://po2.car01.dllstx2.networklayer.com">po2.car01.dllstx2.networklayer.com</a> (70.87.254.78) [AS 21844] 43 msec</div><div> 13 <a href="http://router1-dal.linode.com">router1-dal.linode.com</a> (67.18.7.90) [AS 21844] 42 msec 41 msec 44 msec</div>
</div><div><br></div><div><br></div><div><br></div><div><div>Tracing the route to <a href="http://ge-7-15-19.car1.Atlanta1.Level3.net">ge-7-15-19.car1.Atlanta1.Level3.net</a> (4.69.151.49)</div><div><br></div><div> 1 66.6.235.11 2 msec 2 msec 2 msec</div>
<div> 2 209.66.88.149 [AS 6461] 2 msec 1 msec 2 msec</div><div> 3 <a href="http://xe-3-3-0.cr2.lga5.us.above.net">xe-3-3-0.cr2.lga5.us.above.net</a> (64.125.28.218) [AS 6461] [MPLS: Label 769773 Exp 0] 2 msec 2 msec 1 msec</div>
<div> 4 <a href="http://xe-2-2-0.cr2.dca2.us.above.net">xe-2-2-0.cr2.dca2.us.above.net</a> (64.125.26.105) [AS 6461] [MPLS: Label 590170 Exp 0] 47 msec 6 msec 6 msec</div><div> 5 <a href="http://xe-1-0-1.er2.iad10.us.above.net">xe-1-0-1.er2.iad10.us.above.net</a> (64.125.26.242) [AS 6461] 6 msec 7 msec 7 msec</div>
<div> 6 <a href="http://above-level3.iad10.us.above.net">above-level3.iad10.us.above.net</a> (64.125.13.234) [AS 6461] 33 msec 32 msec 32 msec</div><div> 7 <a href="http://vlan60.csw1.Washington1.Level3.net">vlan60.csw1.Washington1.Level3.net</a> (4.69.149.62) [AS 3356] 105 msec</div>
<div> <a href="http://vlan70.csw2.Washington1.Level3.net">vlan70.csw2.Washington1.Level3.net</a> (4.69.149.126) [AS 3356] 45 msec *</div><div> 8 <a href="http://ae-61-61.ebr1.Washington1.Level3.net">ae-61-61.ebr1.Washington1.Level3.net</a> (4.69.134.129) [AS 3356] 31 msec</div>
<div> <a href="http://ae-81-81.ebr1.Washington1.Level3.net">ae-81-81.ebr1.Washington1.Level3.net</a> (4.69.134.137) [AS 3356] 20 msec</div><div> <a href="http://ae-71-71.ebr1.Washington1.Level3.net">ae-71-71.ebr1.Washington1.Level3.net</a> (4.69.134.133) [AS 3356] 26 msec</div>
<div> 9 <a href="http://ae-2-2.ebr3.Atlanta2.Level3.net">ae-2-2.ebr3.Atlanta2.Level3.net</a> (4.69.132.85) [AS 3356] 51 msec 51 msec 50 msec</div><div> 10 4.69.148.242 [AS 3356] 125 msec</div><div> <a href="http://ae-73-73.ebr2.Atlanta2.Level3.net">ae-73-73.ebr2.Atlanta2.Level3.net</a> (4.69.148.254) [AS 3356] 53 msec 48 msec</div>
<div> 11 <a href="http://ae-11-51.car1.Atlanta1.Level3.net">ae-11-51.car1.Atlanta1.Level3.net</a> (4.69.150.3) [AS 3356] 61 msec * 44 msec</div></div><div><br></div><div><br></div><div>Looking glass trace from Level 3:</div>
<div><br></div><div><pre><font face="terminal" size="1" color="#000000"> 1 <a href="http://ge-6-17-17.car1.Atlanta1.Level3.net">ge-6-17-17.car1.Atlanta1.Level3.net</a> (4.69.151.45) 12 msec
<a href="http://ge-6-13-18.car2.Atlanta1.Level3.net">ge-6-13-18.car2.Atlanta1.Level3.net</a> (4.69.151.73) 16 msec
<a href="http://ge-6-17-17.car1.Atlanta1.Level3.net">ge-6-17-17.car1.Atlanta1.Level3.net</a> (4.69.151.45) 12 msec
2 <a href="http://ae-2-52.edge5.Atlanta2.Level3.net">ae-2-52.edge5.Atlanta2.Level3.net</a> (4.69.150.78) 28 msec * *
3 64.208.27.209 [AS3549 {GBLX}] 12 msec 12 msec 16 msec
4 67.16.131.134 [AS3549 {GBLX}] 60 msec 64 msec
<a href="http://pos3-0-0-2488M.asr1.JFK1.gblx.net">pos3-0-0-2488M.asr1.JFK1.gblx.net</a> (67.16.131.138) [AS3549 {GBLX}] 128 msec
5 159.63.23.214 [AS3549 {GBLX}] 64 msec 60 msec 128 msec
6 * *
66.6.224.47 [AS40073 {NBA-ASN-USA}] !filtered </font></pre></div><div><br></div><div><font color="#c0c0c0" size="1">-- Webmail</font></div><br>
</div>