[outages] Level3 Looping Traffic in LAX
Jeremy Chadwick
jdc at koitsu.org
Sat Dec 13 02:19:29 EST 2014
Few things to say here:
1) I can confirm the behaviour from Comcast, whose packtes in this case
are traversing a different L3 router.
Host Loss% Snt Rcv Last Avg Best Wrst
1. gw.home.lan (192.168.1.1) 0.0% 12 12 0.2 0.2 0.2 0.3
2. 69.181.136.1 0.0% 11 11 8.6 9.4 8.0 13.2
3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69) 0.0% 11 11 8.9 9.9 8.4 13.1
4. te-1-1-0-8-ar01.sfsutro.ca.sfba.comcast.net (68.85.155.146) 0.0% 11 11 11.4 13.5 10.3 16.9
5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93) 0.0% 11 11 14.6 14.5 12.1 19.6
6. 50.242.148.34 0.0% 11 11 11.5 12.5 10.9 16.1
7. ae-2-70.edge5.LosAngeles1.Level3.net (4.69.144.75) 20.0% 11 8 20.7 22.5 19.6 31.7
8. ae-2-70.edge5.LosAngeles1.Level3.net (4.69.144.75) 80.0% 11 2 22.5 23.5 22.5 24.4
9. ???
2) I can confirm "oddities" from my VPS in SoCal, who tends to have good
peering.
Host Loss% Snt Rcv Last Avg Best Wrst
1. 174.136.108.145 0.0% 6 6 13.2 4.3 0.9 13.2
2. s7.lax.arpnetworks.com (208.79.88.135) 0.0% 6 6 1.2 0.8 0.5 1.2
3. r1.lax.arpnetworks.com (208.79.88.2) 0.0% 6 6 0.7 0.7 0.6 0.8
4. ???
3) Looking through route-views.routeviews.org we see something very
interesting:
route-views>traceroute 1.201.0.39
Type escape sequence to abort.
Tracing the route to 1.201.0.39
VRF info: (vrf in name/id, vrf out name/id)
1 vl-51.uonet1-gw.uoregon.edu (128.223.51.2) [AS 3582] 0 msec 0 msec 1 msec
2 vl-3.uonet9-gw.uoregon.edu (128.223.3.9) [AS 3582] 0 msec 1 msec 0 msec
3 vl-675.core0-gw.pdx.oregon-gigapop.net (198.32.163.217) [AS 4600] 3 msec 3 msec 3 msec
4 xe-1-1-0.691.lsan0.tr-cps.internet2.edu (137.164.131.45) [AS 11164] 25 msec 24 msec 24 msec
5 * * *
6 sig-tech.fe3-1.br01.seo01.pccwbtn.net (63.218.149.86) [AS 3491] !A !A !A
"!A" means "communication with destination network administratively
prohibited", i.e. ICMP type 3 code 9.
4) Using HE's looking glass:
core1.fmt2.he.net> traceroute 1.201.0.39
traceroute to 1.201.0.39 (1.201.0.39), 30 hops max, 60 byte packets
1 64.71.148.109 0.235 ms 0.297 ms 0.277 ms
2 72.52.92.65 5.810 ms 5.798 ms 5.855 ms
3 216.218.134.6 10.130 ms 10.066 ms 10.127 ms
4 112.174.87.253 144.112 ms 144.102 ms 144.086 ms
5 112.174.84.117 155.770 ms 112.174.83.205 158.326 ms 112.174.84.165 158.160 ms
6 112.174.84.234 144.461 ms 112.174.83.218 143.929 ms 144.067 ms
7 121.189.3.46 143.521 ms !X 143.405 ms !X 143.380 ms !X
"!X" means "communication administratively prohibited", i.e. ICMP type 3
code 13. 121.189.3.46 according to ARIN is allocated to Korea Telecom.
This could be an edge router rule blocking probe traffic, not sure.
5) Cogent's looking glass shows similar to HE:
traceroute to 1.201.0.39 (1.201.0.39), 30 hops max, 60 byte packets
1 vl5.mag01.jfk02.atlas.cogentco.com (66.28.3.113) 0.440 ms 0.448 ms
2 te0-17-0-5.ccr41.jfk02.atlas.cogentco.com (154.54.5.226) 0.638 ms 0.641 ms
3 be2518.mpd21.dca01.atlas.cogentco.com (154.54.80.162) 7.466 ms 7.431 ms
4 be2171.ccr42.atl01.atlas.cogentco.com (154.54.31.109) 18.202 ms 18.117 ms
5 be2172.ccr21.iah01.atlas.cogentco.com (154.54.29.18) 32.765 ms 32.954 ms
6 be2065.ccr21.lax01.atlas.cogentco.com (154.54.5.66) 68.002 ms 68.420 ms
7 be2179.ccr23.lax05.atlas.cogentco.com (154.54.41.82) 68.945 ms 70.382 ms
8 38.104.84.186 (38.104.84.186) 92.214 ms 92.229 ms
9 112.174.87.205 (112.174.87.205) 231.498 ms 226.864 ms
10 112.174.84.77 (112.174.84.77) 217.304 ms 112.174.84.97 (112.174.84.97) 219.000 ms
11 112.174.84.122 (112.174.84.122) 227.297 ms 112.174.84.218 (112.174.84.218) 219.319 ms
12 121.189.3.46 (121.189.3.46) 232.136 ms !X 232.086 ms !X
6) And finally, NTT's looking glass:
1 ae-2.r21.atlnga05.us.bb.gin.ntt.net (129.250.2.180) [MPLS: Label 300720 Exp 0] 0 msec 53 msec 58 msec
2 ae-7.r21.dllstx09.us.bb.gin.ntt.net (129.250.3.91) [MPLS: Labels 317513/299888 Exp 0] 19 msec 20 msec 23 msec
3 ae-4.r21.snjsca04.us.bb.gin.ntt.net (129.250.4.25) [MPLS: Labels 473537/299888 Exp 0] 64 msec 60 msec 62 msec
4 ae-4.r21.tokyjp05.jp.bb.gin.ntt.net (129.250.2.19) [MPLS: Labels 325362/299888 Exp 0] 167 msec 159 msec 158 msec
5 ae-0.r24.tokyjp05.jp.bb.gin.ntt.net (129.250.6.96) [MPLS: Label 299824 Exp 0] 169 msec 171 msec 164 msec
6 ae-1.r01.tokyjp03.jp.bb.gin.ntt.net (129.250.6.166) 169 msec 168 msec
ae-2.r01.tokyjp03.jp.bb.gin.ntt.net (129.250.6.170) 176 msec
7 * * *
8 * * *
9 * * *
10 * * *
{repeat until TTL reached}
--
| Jeremy Chadwick jdc at koitsu.org |
| UNIX Systems Administrator http://jdc.koitsu.org/ |
| Making life hard for others since 1977. PGP 4BD6C0CB |
On Fri, Dec 12, 2014 at 10:59:02PM -0800, Ca By via Outages wrote:
> Seems hop 7 and 8 are the same box.... This traceroute works fine from AT&T
> and Sprint, they deliver it correctly to Korea instead of black-holing it
> in LAX
>
> traceroute to 1.201.0.39 (1.201.0.39), 30 hops max, 60 byte packets
> x
> 3 te-0-4-0-21-sur03.seattle.wa.seattle.comcast.net (68.86.113.61) 25.810
> ms 25.771 ms 25.733 ms
> 4 be-20-ar01.seattle.wa.seattle.comcast.net (69.139.164.129) 25.670 ms
> 25.630 ms 25.577 ms
> 5 he-1-3-0-0-10-cr01.seattle.wa.ibone.comcast.net (68.86.93.165) 25.508
> ms 25.473 ms 25.405 ms
> 6 4.68.63.65 (4.68.63.65) 22.755 ms 13.858 ms 16.425 ms
> 7 * * ae-3-80.edge5.LosAngeles1.Level3.net (4.69.144.139) 61.317 ms
> 8 * ae-3-80.edge5.LosAngeles1.Level3.net (4.69.144.139) 58.978 ms
> 58.956 ms
> 9 * * *
> 10 * * *
> 11 * * *
> 12 * * *
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
More information about the Outages
mailing list