[outages] Level 3 Problems

Wed Aug 15 14:37:51 EDT 2012

Level 3 has updated our ticket and is blaming this on a DDoS attack:

This is to confirm that an update has been recorded on Level 3 ticket
5851586.

*** CASCADED EXTERNAL NOTES 15-Aug-2012 18:31:55 GMT hails From CASE:
5851387 - Event
IP Operations Engineers and Security Operations continue to add Access
Lists to block suspected attacking IP addresses and ports on affected
equipment. No estimated time to complete this action has been provided.
Affected customers may continue to experience internet access failures over
the Level 3 network while repairs progress. No estimated time to restore
has been provided.

Dan

-- Webmail

On Wed, Aug 15, 2012 at 11:54 AM, Joseph Jackson
<jjackson at aninetworks.net>wrote:

> We've seen some packet loss on our customer end points out of NYC over
> level3 this morning also.  Hosted at 60 Hudson.****
>
> ** **
>
> ** **
>
> *From:* outages-bounces at outages.org [mailto:outages-bounces at outages.org] *On
> Behalf Of *Dan Stralka
> *Sent:* Wednesday, August 15, 2012 10:15 AM
>
> *To:* outages at outages.org
> *Subject:* [outages] Level 3 Problems****
>
> ** **
>
> Seeing high latency on Level 3 out of northern New Jersey.  We saw very
> slow website loads across many sites and a few page timeouts.  We also see
> high latency between our Abovenet circuit and Level 3 where they peer.
> Level 3 tech support confirms latency problems and did say a lot of
> customers are calling about this. See traceroutes below:****
>
> ** **
>
> Level 3 Router:****
>
> ** **
>
> Type escape sequence to abort.****
>
> Tracing the route to france.com (96.126.117.104)****
>
> VRF info: (vrf in name/id, vrf out name/id)****
>
>   1 159.63.23.213 1 msec 1 msec 1 msec****
>
>   2 64.215.195.230 [AS 3549] 100 msec 103 msec 114 msec****
>
>   3 ae-4-90.edge3.NewYork1.Level3.net (4.69.155.209) [AS 3356] 49 msec 38
> msec****
>
>     ae-3-80.edge3.NewYork1.Level3.net (4.69.155.145) [AS 3356] 35 msec****
>
>   4 te2-5.bbr01.tl01.nyc01.networklayer.com (4.26.34.22) [AS 3356] 99
> msec 97 msec 98 msec****
>
>   5 ae1.bbr01.eq01.chi01.networklayer.com (173.192.18.132) [AS 36351]
> [MPLS: Label 348745 Exp 0] 119 msec *  119 msec****
>
>   6 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) [AS 36351] 77
> msec 76 msec 128 msec****
>
>   7 po31.dsr01.dllstx3.networklayer.com (173.192.18.225) [AS 36351] 76
> msec 79 msec 80 msec****
>
>   8 te2-4.dsr01.dllstx2.networklayer.com (70.87.255.110) [AS 21844]
> [MPLS: Label 991 Exp 0] 73 msec 69 msec 72 msec****
>
>   9 po1.car01.dllstx2.networklayer.com (70.87.254.74) [AS 21844] 76 msec
> 76 msec 82 msec****
>
>  10 router1-dal.linode.com (67.18.7.90) [AS 21844] 137 msec 142 msec *****
>
> ** **
>
> ** **
>
> NJASR1#trace 4.69.151.49****
>
> Type escape sequence to abort.****
>
> Tracing the route to ge-7-15-19.car1.Atlanta1.Level3.net (4.69.151.49)****
>
> VRF info: (vrf in name/id, vrf out name/id)****
>
>   1 159.63.23.213 1 msec 1 msec 1 msec****
>
>   2 64.215.195.230 [AS 3549] 94 msec *  *****
>
>   3 vlan60.csw1.NewYork1.Level3.net (4.69.155.62) [AS 3356] 34 msec *  36
> msec****
>
>   4 ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) [AS 3356] 30 msec****
>
>     ae-61-61.ebr1.NewYork1.Level3.net (4.69.134.65) [AS 3356] 100 msec****
>
>     ae-71-71.ebr1.NewYork1.Level3.net (4.69.134.69) [AS 3356] 34 msec****
>
>   5  *****
>
>     ae-10-10.ebr2.Washington12.Level3.net (4.69.148.50) [AS 3356] 36 msec
> 46 msec****
>
>   6  *****
>
>     ae-1-100.ebr1.Washington12.Level3.net (4.69.143.213) [AS 3356] 122
> msec 108 msec****
>
>   7 ae-6-6.ebr1.Atlanta2.Level3.net (4.69.148.105) [AS 3356] 132 msec *  *
> ****
>
>   8 ae-11-51.car1.Atlanta1.Level3.net (4.69.150.3) [AS 3356] 23 msec *
>  23 msec****
>
> ** **
>
> ** **
>
> ** **
>
> ** **
>
> Abovenet (for comparison):****
>
> ** **
>
> ** **
>
> Type escape sequence to abort.****
>
> Tracing the route to france.com (96.126.117.104)****
>
> ** **
>
>   1 209.66.88.145 1 msec 1 msec 1 msec****
>
>   2 xe-4-1-0.cr1.lga5.us.above.net (64.125.31.170) [AS 6461] [MPLS: Label
> 677655 Exp 0] 1 msec 1 msec 1 msec****
>
>   3 xe-1-0-1.er3.lga5.us.above.net (64.125.24.165) [AS 6461] 1 msec 1
> msec 1 msec****
>
>   4 above-ntt-2.lga5.us.above.net (64.125.13.182) [AS 6461] 1 msec 1 msec
> 1 msec****
>
>   5 ae-3.r05.nycmny01.us.bb.gin.ntt.net (129.250.4.204) [AS 2914] 2 msec
> 1 msec 2 msec****
>
>   6 xe-1-0-0.bbr01.tl01.nyc01.networklayer.com (165.254.16.2) [AS 2914] 1
> msec 1 msec 1 msec****
>
>   7 ae7.bbr02.tl01.nyc01.networklayer.com (173.192.18.177) [AS 36351]
> [MPLS: Label 400085 Exp 0] 1 msec 2 msec 1 msec****
>
>   8 ae1.bbr01.eq01.chi01.networklayer.com (173.192.18.132) [AS 36351]
> [MPLS: Label 347833 Exp 0] 20 msec 40 msec 34 msec****
>
>   9 ae20.bbr01.eq01.dal03.networklayer.com (173.192.18.136) [AS 36351] 44
> msec 42 msec 45 msec****
>
>  10 po31.dsr02.dllstx3.networklayer.com (173.192.18.227) [AS 36351] 42
> msec****
>
>     po31.dsr01.dllstx3.networklayer.com (173.192.18.225) [AS 36351] 40
> msec 41 msec****
>
>  11 te2-4.dsr02.dllstx2.networklayer.com (70.87.255.126) [AS 21844]
> [MPLS: Label 504 Exp 0] 45 msec****
>
>     te2-4.dsr01.dllstx2.networklayer.com (70.87.255.110) [AS 21844]
> [MPLS: Label 991 Exp 0] 44 msec 40 msec****
>
>  12 po2.car01.dllstx2.networklayer.com (70.87.254.78) [AS 21844] 40 msec**
> **
>
>     po1.car01.dllstx2.networklayer.com (70.87.254.74) [AS 21844] 44 msec**
> **
>
>     po2.car01.dllstx2.networklayer.com (70.87.254.78) [AS 21844] 43 msec**
> **
>
>  13 router1-dal.linode.com (67.18.7.90) [AS 21844] 42 msec 41 msec 44 msec
> ****
>
> ** **
>
> ** **
>
> ** **
>
> Tracing the route to ge-7-15-19.car1.Atlanta1.Level3.net (4.69.151.49)****
>
> ** **
>
>   1 66.6.235.11 2 msec 2 msec 2 msec****
>
>   2 209.66.88.149 [AS 6461] 2 msec 1 msec 2 msec****
>
>   3 xe-3-3-0.cr2.lga5.us.above.net (64.125.28.218) [AS 6461] [MPLS: Label
> 769773 Exp 0] 2 msec 2 msec 1 msec****
>
>   4 xe-2-2-0.cr2.dca2.us.above.net (64.125.26.105) [AS 6461] [MPLS: Label
> 590170 Exp 0] 47 msec 6 msec 6 msec****
>
>   5 xe-1-0-1.er2.iad10.us.above.net (64.125.26.242) [AS 6461] 6 msec 7
> msec 7 msec****
>
>   6 above-level3.iad10.us.above.net (64.125.13.234) [AS 6461] 33 msec 32
> msec 32 msec****
>
>   7 vlan60.csw1.Washington1.Level3.net (4.69.149.62) [AS 3356] 105 msec***
> *
>
>     vlan70.csw2.Washington1.Level3.net (4.69.149.126) [AS 3356] 45 msec **
> ***
>
>   8 ae-61-61.ebr1.Washington1.Level3.net (4.69.134.129) [AS 3356] 31 msec*
> ***
>
>     ae-81-81.ebr1.Washington1.Level3.net (4.69.134.137) [AS 3356] 20 msec*
> ***
>
>     ae-71-71.ebr1.Washington1.Level3.net (4.69.134.133) [AS 3356] 26 msec*
> ***
>
>   9 ae-2-2.ebr3.Atlanta2.Level3.net (4.69.132.85) [AS 3356] 51 msec 51
> msec 50 msec****
>
>  10 4.69.148.242 [AS 3356] 125 msec****
>
>     ae-73-73.ebr2.Atlanta2.Level3.net (4.69.148.254) [AS 3356] 53 msec 48
> msec****
>
>  11 ae-11-51.car1.Atlanta1.Level3.net (4.69.150.3) [AS 3356] 61 msec *
>  44 msec****
>
> ** **
>
> ** **
>
> Looking glass trace from Level 3:****
>
> ** **
>
>   1 ge-6-17-17.car1.Atlanta1.Level3.net (4.69.151.45) 12 msec****
>
>     ge-6-13-18.car2.Atlanta1.Level3.net (4.69.151.73) 16 msec****
>
>     ge-6-17-17.car1.Atlanta1.Level3.net (4.69.151.45) 12 msec****
>
>   2 ae-2-52.edge5.Atlanta2.Level3.net (4.69.150.78) 28 msec *  * ****
>
>   3 64.208.27.209 [AS3549 {GBLX}] 12 msec 12 msec 16 msec****
>
>   4 67.16.131.134 [AS3549 {GBLX}] 60 msec 64 msec****
>
>     pos3-0-0-2488M.asr1.JFK1.gblx.net (67.16.131.138) [AS3549 {GBLX}] 128 msec****
>
>   5 159.63.23.214 [AS3549 {GBLX}] 64 msec 60 msec 128 msec****
>
>   6  *  * ****
>
>     66.6.224.47 [AS40073 {NBA-ASN-USA}] !filtered ****
>
> ** **
>
> -- Webmail****
>
> ** **
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20120815/17a80511/attachment.htm>