Level 3 has updated our ticket and is blaming this on a DDoS attack:<div><br></div><div><br></div><div>This is to confirm that an update has been recorded on Level 3 ticket
5851586.<br><br><br>*** CASCADED EXTERNAL NOTES 15-Aug-2012 18:31:55 GMT hails
>From CASE: 5851387 - Event<br>IP Operations Engineers and Security Operations
continue to add Access Lists to block suspected attacking IP addresses and ports
on affected equipment. No estimated time to complete this action has been
provided. Affected customers may continue to experience internet access failures
over the Level 3 network while repairs progress. No estimated time to restore
has been provided.</div><div><br></div><div><br>Dan<div><br></div><div><font color="#c0c0c0" size="1">-- Webmail</font></div><br>
<br><br><div class="gmail_quote">On Wed, Aug 15, 2012 at 11:54 AM, Joseph Jackson <span dir="ltr"><<a href="mailto:jjackson@aninetworks.net" target="_blank">jjackson@aninetworks.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We've seen some packet loss on our customer end points out of NYC over level3 this morning also. Hosted at 60 Hudson.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in"><p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> <a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a> [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>] <b>On Behalf Of </b>Dan Stralka<br>
<b>Sent:</b> Wednesday, August 15, 2012 10:15 AM</span></p><div class="im"><br><b>To:</b> <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br><b>Subject:</b> [outages] Level 3 Problems<u></u><u></u></div>
<p></p></div><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Seeing high latency on Level 3 out of northern New Jersey. We saw very slow website loads across many sites and a few page timeouts. We also see high latency between our Abovenet circuit and Level 3 where they peer. Level 3 tech support confirms latency problems and did say a lot of customers are calling about this. See traceroutes below:<u></u><u></u></p>
<div><div class="h5"><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Level 3 Router:<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><div><p class="MsoNormal">Type escape sequence to abort.<u></u><u></u></p>
</div><div><p class="MsoNormal">Tracing the route to <a href="http://france.com" target="_blank">france.com</a> (96.126.117.104)<u></u><u></u></p></div><div><p class="MsoNormal">VRF info: (vrf in name/id, vrf out name/id)<u></u><u></u></p>
</div><div><p class="MsoNormal"> 1 159.63.23.213 1 msec 1 msec 1 msec<u></u><u></u></p></div><div><p class="MsoNormal"> 2 64.215.195.230 [AS 3549] 100 msec 103 msec 114 msec<u></u><u></u></p></div><div><p class="MsoNormal">
3 <a href="http://ae-4-90.edge3.NewYork1.Level3.net" target="_blank">ae-4-90.edge3.NewYork1.Level3.net</a> (4.69.155.209) [AS 3356] 49 msec 38 msec<u></u><u></u></p></div><div><p class="MsoNormal"> <a href="http://ae-3-80.edge3.NewYork1.Level3.net" target="_blank">ae-3-80.edge3.NewYork1.Level3.net</a> (4.69.155.145) [AS 3356] 35 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 4 <a href="http://te2-5.bbr01.tl01.nyc01.networklayer.com" target="_blank">te2-5.bbr01.tl01.nyc01.networklayer.com</a> (4.26.34.22) [AS 3356] 99 msec 97 msec 98 msec<u></u><u></u></p></div>
<div><p class="MsoNormal"> 5 <a href="http://ae1.bbr01.eq01.chi01.networklayer.com" target="_blank">ae1.bbr01.eq01.chi01.networklayer.com</a> <a href="tel:%28173.192.18.132" value="+17319218132" target="_blank">(173.192.18.132</a>) [AS 36351] [MPLS: Label 348745 Exp 0] 119 msec * 119 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 6 <a href="http://ae20.bbr01.eq01.dal03.networklayer.com" target="_blank">ae20.bbr01.eq01.dal03.networklayer.com</a> <a href="tel:%28173.192.18.136" value="+17319218136" target="_blank">(173.192.18.136</a>) [AS 36351] 77 msec 76 msec 128 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 7 <a href="http://po31.dsr01.dllstx3.networklayer.com" target="_blank">po31.dsr01.dllstx3.networklayer.com</a> <a href="tel:%28173.192.18.225" value="+17319218225" target="_blank">(173.192.18.225</a>) [AS 36351] 76 msec 79 msec 80 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 8 <a href="http://te2-4.dsr01.dllstx2.networklayer.com" target="_blank">te2-4.dsr01.dllstx2.networklayer.com</a> <a href="tel:%2870.87.255.110" value="+17087255110" target="_blank">(70.87.255.110</a>) [AS 21844] [MPLS: Label 991 Exp 0] 73 msec 69 msec 72 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 9 <a href="http://po1.car01.dllstx2.networklayer.com" target="_blank">po1.car01.dllstx2.networklayer.com</a> (70.87.254.74) [AS 21844] 76 msec 76 msec 82 msec<u></u><u></u></p></div><div>
<p class="MsoNormal"> 10 <a href="http://router1-dal.linode.com" target="_blank">router1-dal.linode.com</a> (67.18.7.90) [AS 21844] 137 msec 142 msec *<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div>
<div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><div><p class="MsoNormal">NJASR1#trace 4.69.151.49<u></u><u></u></p></div><div><p class="MsoNormal">Type escape sequence to abort.<u></u><u></u></p></div><div><p class="MsoNormal">
Tracing the route to <a href="http://ge-7-15-19.car1.Atlanta1.Level3.net" target="_blank">ge-7-15-19.car1.Atlanta1.Level3.net</a> (4.69.151.49)<u></u><u></u></p></div><div><p class="MsoNormal">VRF info: (vrf in name/id, vrf out name/id)<u></u><u></u></p>
</div><div><p class="MsoNormal"> 1 159.63.23.213 1 msec 1 msec 1 msec<u></u><u></u></p></div><div><p class="MsoNormal"> 2 64.215.195.230 [AS 3549] 94 msec * *<u></u><u></u></p></div><div><p class="MsoNormal"> 3 <a href="http://vlan60.csw1.NewYork1.Level3.net" target="_blank">vlan60.csw1.NewYork1.Level3.net</a> (4.69.155.62) [AS 3356] 34 msec * 36 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 4 <a href="http://ae-71-71.ebr1.NewYork1.Level3.net" target="_blank">ae-71-71.ebr1.NewYork1.Level3.net</a> (4.69.134.69) [AS 3356] 30 msec<u></u><u></u></p></div><div><p class="MsoNormal">
<a href="http://ae-61-61.ebr1.NewYork1.Level3.net" target="_blank">ae-61-61.ebr1.NewYork1.Level3.net</a> (4.69.134.65) [AS 3356] 100 msec<u></u><u></u></p></div><div><p class="MsoNormal"> <a href="http://ae-71-71.ebr1.NewYork1.Level3.net" target="_blank">ae-71-71.ebr1.NewYork1.Level3.net</a> (4.69.134.69) [AS 3356] 34 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 5 *<u></u><u></u></p></div><div><p class="MsoNormal"> <a href="http://ae-10-10.ebr2.Washington12.Level3.net" target="_blank">ae-10-10.ebr2.Washington12.Level3.net</a> (4.69.148.50) [AS 3356] 36 msec 46 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 6 *<u></u><u></u></p></div><div><p class="MsoNormal"> <a href="http://ae-1-100.ebr1.Washington12.Level3.net" target="_blank">ae-1-100.ebr1.Washington12.Level3.net</a> (4.69.143.213) [AS 3356] 122 msec 108 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 7 <a href="http://ae-6-6.ebr1.Atlanta2.Level3.net" target="_blank">ae-6-6.ebr1.Atlanta2.Level3.net</a> (4.69.148.105) [AS 3356] 132 msec * *<u></u><u></u></p></div><div><p class="MsoNormal">
8 <a href="http://ae-11-51.car1.Atlanta1.Level3.net" target="_blank">ae-11-51.car1.Atlanta1.Level3.net</a> (4.69.150.3) [AS 3356] 23 msec * 23 msec<u></u><u></u></p></div></div></div><div><p class="MsoNormal"><u></u> <u></u></p>
</div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Abovenet (for comparison):<u></u><u></u></p>
</div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><div><p class="MsoNormal">Type escape sequence to abort.<u></u><u></u></p></div><div><p class="MsoNormal">Tracing the route to <a href="http://france.com" target="_blank">france.com</a> (96.126.117.104)<u></u><u></u></p>
</div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"> <a href="tel:1%20209.66.88.145" value="+12096688145" target="_blank">1 209.66.88.145</a> 1 msec 1 msec 1 msec<u></u><u></u></p></div><div>
<p class="MsoNormal"> 2 <a href="http://xe-4-1-0.cr1.lga5.us.above.net" target="_blank">xe-4-1-0.cr1.lga5.us.above.net</a> <a href="tel:%2864.125.31.170" value="+16412531170" target="_blank">(64.125.31.170</a>) [AS 6461] [MPLS: Label 677655 Exp 0] 1 msec 1 msec 1 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 3 <a href="http://xe-1-0-1.er3.lga5.us.above.net" target="_blank">xe-1-0-1.er3.lga5.us.above.net</a> <a href="tel:%2864.125.24.165" value="+16412524165" target="_blank">(64.125.24.165</a>) [AS 6461] 1 msec 1 msec 1 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 4 <a href="http://above-ntt-2.lga5.us.above.net" target="_blank">above-ntt-2.lga5.us.above.net</a> <a href="tel:%2864.125.13.182" value="+16412513182" target="_blank">(64.125.13.182</a>) [AS 6461] 1 msec 1 msec 1 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 5 <a href="http://ae-3.r05.nycmny01.us.bb.gin.ntt.net" target="_blank">ae-3.r05.nycmny01.us.bb.gin.ntt.net</a> (129.250.4.204) [AS 2914] 2 msec 1 msec 2 msec<u></u><u></u></p></div><div><p class="MsoNormal">
6 <a href="http://xe-1-0-0.bbr01.tl01.nyc01.networklayer.com" target="_blank">xe-1-0-0.bbr01.tl01.nyc01.networklayer.com</a> (165.254.16.2) [AS 2914] 1 msec 1 msec 1 msec<u></u><u></u></p></div><div><p class="MsoNormal">
7 <a href="http://ae7.bbr02.tl01.nyc01.networklayer.com" target="_blank">ae7.bbr02.tl01.nyc01.networklayer.com</a> <a href="tel:%28173.192.18.177" value="+17319218177" target="_blank">(173.192.18.177</a>) [AS 36351] [MPLS: Label 400085 Exp 0] 1 msec 2 msec 1 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 8 <a href="http://ae1.bbr01.eq01.chi01.networklayer.com" target="_blank">ae1.bbr01.eq01.chi01.networklayer.com</a> <a href="tel:%28173.192.18.132" value="+17319218132" target="_blank">(173.192.18.132</a>) [AS 36351] [MPLS: Label 347833 Exp 0] 20 msec 40 msec 34 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 9 <a href="http://ae20.bbr01.eq01.dal03.networklayer.com" target="_blank">ae20.bbr01.eq01.dal03.networklayer.com</a> <a href="tel:%28173.192.18.136" value="+17319218136" target="_blank">(173.192.18.136</a>) [AS 36351] 44 msec 42 msec 45 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 10 <a href="http://po31.dsr02.dllstx3.networklayer.com" target="_blank">po31.dsr02.dllstx3.networklayer.com</a> (173.192.18.227) [AS 36351] 42 msec<u></u><u></u></p></div><div><p class="MsoNormal">
<a href="http://po31.dsr01.dllstx3.networklayer.com" target="_blank">po31.dsr01.dllstx3.networklayer.com</a> (173.192.18.225) [AS 36351] 40 msec 41 msec<u></u><u></u></p></div><div><p class="MsoNormal"> 11 <a href="http://te2-4.dsr02.dllstx2.networklayer.com" target="_blank">te2-4.dsr02.dllstx2.networklayer.com</a> (70.87.255.126) [AS 21844] [MPLS: Label 504 Exp 0] 45 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> <a href="http://te2-4.dsr01.dllstx2.networklayer.com" target="_blank">te2-4.dsr01.dllstx2.networklayer.com</a> (70.87.255.110) [AS 21844] [MPLS: Label 991 Exp 0] 44 msec 40 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 12 <a href="http://po2.car01.dllstx2.networklayer.com" target="_blank">po2.car01.dllstx2.networklayer.com</a> (70.87.254.78) [AS 21844] 40 msec<u></u><u></u></p></div><div><p class="MsoNormal">
<a href="http://po1.car01.dllstx2.networklayer.com" target="_blank">po1.car01.dllstx2.networklayer.com</a> (70.87.254.74) [AS 21844] 44 msec<u></u><u></u></p></div><div><p class="MsoNormal"> <a href="http://po2.car01.dllstx2.networklayer.com" target="_blank">po2.car01.dllstx2.networklayer.com</a> (70.87.254.78) [AS 21844] 43 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 13 <a href="http://router1-dal.linode.com" target="_blank">router1-dal.linode.com</a> (67.18.7.90) [AS 21844] 42 msec 41 msec 44 msec<u></u><u></u></p></div></div><div><p class="MsoNormal">
<u></u> <u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><div><p class="MsoNormal">Tracing the route to <a href="http://ge-7-15-19.car1.Atlanta1.Level3.net" target="_blank">ge-7-15-19.car1.Atlanta1.Level3.net</a> (4.69.151.49)<u></u><u></u></p>
</div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal"> 1 66.6.235.11 2 msec 2 msec 2 msec<u></u><u></u></p></div><div><p class="MsoNormal"> 2 209.66.88.149 [AS 6461] 2 msec 1 msec 2 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 3 <a href="http://xe-3-3-0.cr2.lga5.us.above.net" target="_blank">xe-3-3-0.cr2.lga5.us.above.net</a> (64.125.28.218) [AS 6461] [MPLS: Label 769773 Exp 0] 2 msec 2 msec 1 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 4 <a href="http://xe-2-2-0.cr2.dca2.us.above.net" target="_blank">xe-2-2-0.cr2.dca2.us.above.net</a> (64.125.26.105) [AS 6461] [MPLS: Label 590170 Exp 0] 47 msec 6 msec 6 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> 5 <a href="http://xe-1-0-1.er2.iad10.us.above.net" target="_blank">xe-1-0-1.er2.iad10.us.above.net</a> (64.125.26.242) [AS 6461] 6 msec 7 msec 7 msec<u></u><u></u></p></div><div><p class="MsoNormal">
6 <a href="http://above-level3.iad10.us.above.net" target="_blank">above-level3.iad10.us.above.net</a> (64.125.13.234) [AS 6461] 33 msec 32 msec 32 msec<u></u><u></u></p></div><div><p class="MsoNormal"> 7 <a href="http://vlan60.csw1.Washington1.Level3.net" target="_blank">vlan60.csw1.Washington1.Level3.net</a> (4.69.149.62) [AS 3356] 105 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> <a href="http://vlan70.csw2.Washington1.Level3.net" target="_blank">vlan70.csw2.Washington1.Level3.net</a> (4.69.149.126) [AS 3356] 45 msec *<u></u><u></u></p></div><div><p class="MsoNormal">
8 <a href="http://ae-61-61.ebr1.Washington1.Level3.net" target="_blank">ae-61-61.ebr1.Washington1.Level3.net</a> (4.69.134.129) [AS 3356] 31 msec<u></u><u></u></p></div><div><p class="MsoNormal"> <a href="http://ae-81-81.ebr1.Washington1.Level3.net" target="_blank">ae-81-81.ebr1.Washington1.Level3.net</a> (4.69.134.137) [AS 3356] 20 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> <a href="http://ae-71-71.ebr1.Washington1.Level3.net" target="_blank">ae-71-71.ebr1.Washington1.Level3.net</a> (4.69.134.133) [AS 3356] 26 msec<u></u><u></u></p></div><div><p class="MsoNormal">
9 <a href="http://ae-2-2.ebr3.Atlanta2.Level3.net" target="_blank">ae-2-2.ebr3.Atlanta2.Level3.net</a> (4.69.132.85) [AS 3356] 51 msec 51 msec 50 msec<u></u><u></u></p></div><div><p class="MsoNormal"> 10 4.69.148.242 [AS 3356] 125 msec<u></u><u></u></p>
</div><div><p class="MsoNormal"> <a href="http://ae-73-73.ebr2.Atlanta2.Level3.net" target="_blank">ae-73-73.ebr2.Atlanta2.Level3.net</a> (4.69.148.254) [AS 3356] 53 msec 48 msec<u></u><u></u></p></div><div><p class="MsoNormal">
11 <a href="http://ae-11-51.car1.Atlanta1.Level3.net" target="_blank">ae-11-51.car1.Atlanta1.Level3.net</a> (4.69.150.3) [AS 3356] 61 msec * 44 msec<u></u><u></u></p></div></div><div><p class="MsoNormal"><u></u> <u></u></p>
</div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Looking glass trace from Level 3:<u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><pre><span style="font-size:7.5pt;font-family:"terminal","serif""> 1 <a href="http://ge-6-17-17.car1.Atlanta1.Level3.net" target="_blank">ge-6-17-17.car1.Atlanta1.Level3.net</a> (4.69.151.45) 12 msec<u></u><u></u></span></pre>
<pre><span style="font-size:7.5pt;font-family:"terminal","serif""> <a href="http://ge-6-13-18.car2.Atlanta1.Level3.net" target="_blank">ge-6-13-18.car2.Atlanta1.Level3.net</a> (4.69.151.73) 16 msec<u></u><u></u></span></pre>
<pre><span style="font-size:7.5pt;font-family:"terminal","serif""> <a href="http://ge-6-17-17.car1.Atlanta1.Level3.net" target="_blank">ge-6-17-17.car1.Atlanta1.Level3.net</a> (4.69.151.45) 12 msec<u></u><u></u></span></pre>
<pre><span style="font-size:7.5pt;font-family:"terminal","serif""> 2 <a href="http://ae-2-52.edge5.Atlanta2.Level3.net" target="_blank">ae-2-52.edge5.Atlanta2.Level3.net</a> (4.69.150.78) 28 msec * * <u></u><u></u></span></pre>
<pre><span style="font-size:7.5pt;font-family:"terminal","serif""> 3 64.208.27.209 [AS3549 {GBLX}] 12 msec 12 msec 16 msec<u></u><u></u></span></pre><pre><span style="font-size:7.5pt;font-family:"terminal","serif""> 4 67.16.131.134 [AS3549 {GBLX}] 60 msec 64 msec<u></u><u></u></span></pre>
<pre><span style="font-size:7.5pt;font-family:"terminal","serif""> <a href="http://pos3-0-0-2488M.asr1.JFK1.gblx.net" target="_blank">pos3-0-0-2488M.asr1.JFK1.gblx.net</a> (67.16.131.138) [AS3549 {GBLX}] 128 msec<u></u><u></u></span></pre>
<pre><span style="font-size:7.5pt;font-family:"terminal","serif""> 5 159.63.23.214 [AS3549 {GBLX}] 64 msec 60 msec 128 msec<u></u><u></u></span></pre><pre><span style="font-size:7.5pt;font-family:"terminal","serif""> 6 * * <u></u><u></u></span></pre>
<pre><span style="font-size:7.5pt;font-family:"terminal","serif""> 66.6.224.47 [AS40073 {NBA-ASN-USA}] !filtered </span><u></u><u></u></pre></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div>
<p class="MsoNormal"><span style="font-size:7.5pt;color:silver">-- Webmail</span><u></u><u></u></p></div><p class="MsoNormal"><u></u> <u></u></p></div></div></div></div></div></blockquote></div><br></div>