[outages] LAX routing anomaly: NetworkLayer? CoreSite? Hong Kong ISPs?
Jeremy Chadwick
jdc at koitsu.org
Sun Sep 29 22:31:21 EDT 2013
I just came across this on NANOG while troubleshooting something:
http://mailman.nanog.org/pipermail/nanog/2013-September/061332.html
If the issue described there is what I'm seeing, then it looks like
some part of the 'net in the LA area is very very broken -- and worse,
*has* been broken since roughly Sep 27 21:16:00 PDT (UTC-7).
What I saw (and am still seeing):
src IP: 208.79.90.130 (Southern CA, AS 25795 (I think))
dst IP: 67.18.187.25 (Texas, AS 21844 (I think))
=== Fri Sep 27 21:14:00 PDT 2013 (1380341640)
Start: Fri Sep 27 21:14:00 2013
HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst
1.|-- 208.79.90.129 0.0% 40 40 3.7 11.0 0.9 202.9
2.|-- 206.223.143.131 0.0% 40 40 0.6 0.8 0.5 8.5
3.|-- 173.192.18.140 2.5% 40 39 28.4 30.1 28.2 86.8
4.|-- 173.192.18.225 0.0% 40 40 29.0 29.2 28.7 40.9
5.|-- 70.87.255.66 0.0% 40 40 31.6 31.5 31.3 31.9
6.|-- 70.87.254.74 0.0% 40 40 29.0 31.5 28.6 60.3
7.|-- 67.18.7.90 0.0% 40 40 28.7 28.6 28.4 29.0
8.|-- 67.18.187.25 0.0% 40 40 31.9 31.5 31.3 31.9
=== END
=== Fri Sep 27 21:15:00 PDT 2013 (1380341700)
Start: Fri Sep 27 21:15:00 2013
HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst
1.|-- 208.79.90.129 0.0% 40 40 2.2 51.1 0.9 245.7
2.|-- 206.223.143.131 2.5% 40 39 0.9 12.3 0.5 183.2
3.|-- 173.192.18.140 5.0% 40 38 0.8 29.4 0.7 284.1
4.|-- 173.192.18.225 0.0% 40 40 0.8 17.1 0.5 177.8
5.|-- 70.87.255.66 30.0% 40 28 11.2 39.1 3.4 199.4
6.|-- 70.87.254.74 0.0% 40 40 11.0 21.1 0.5 121.5
7.|-- 67.18.7.90 0.0% 40 40 163.8 122.6 4.6 347.0
8.|-- 67.18.187.25 22.5% 40 31 171.6 81.4 0.5 173.8
=== END
=== Fri Sep 27 21:16:00 PDT 2013 (1380341760)
Start: Fri Sep 27 21:16:00 2013
HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst
1.|-- 208.79.90.129 0.0% 40 40 2.9 7.4 0.9 95.8
2.|-- 208.79.88.135 0.0% 40 40 0.6 15.8 0.4 182.3
3.|-- 129.250.198.185 0.0% 40 40 0.7 0.8 0.7 1.2
4.|-- 129.250.5.69 0.0% 40 40 0.7 5.1 0.6 34.1
5.|-- 129.250.6.11 45.0% 40 22 17.8 14.6 9.6 47.1
6.|-- 129.250.5.53 0.0% 40 40 11.6 10.8 10.0 12.2
7.|-- 128.241.219.234 0.0% 40 40 162.7 163.1 161.7 178.3
8.|-- 173.192.18.151 2.5% 40 39 172.3 172.3 171.2 176.3
9.|-- 173.192.18.166 12.5% 40 35 157.2 156.3 155.6 157.9
10.|-- 173.192.18.140 7.5% 40 37 198.5 201.1 198.4 234.7
11.|-- 173.192.18.225 0.0% 40 40 183.0 190.1 182.9 355.2
12.|-- 70.87.255.66 0.0% 40 40 183.0 183.8 183.0 185.9
13.|-- 70.87.254.74 0.0% 40 40 193.6 195.8 192.0 284.1
14.|-- 67.18.7.90 2.5% 40 39 197.0 197.4 196.9 198.8
15.|-- 67.18.187.25 2.5% 40 39 192.8 192.8 192.2 194.0
=== END
FYI: 206.223.143.131 resolves to te2-6.bbr01.cs01.lax01.networklayer.com.any2ix.coresite.com.
ARIN states 206.223.143.0/24 is CoreSite.
The packet path for the above is:
LAX (ARP Networks)
-> SFO (NTT/Verio)
-> San Jose (NTT/Verio)
-> LAX (NetworkLayer)
-> Dallas (NetworkLayer and Linode)
-> 67.18.187.25
And now for the amusing part -- the return path:
src IP: 67.18.187.25 (Texas, AS 21844 (I think))
dst IP: 208.79.90.130 (Southern CA, AS 25795 (I think))
traceroute to omake.koitsu.org (208.79.90.130), 30 hops max, 60 byte packets
1 router2-dal.linode.com (67.18.7.162) 0.604 ms 0.812 ms 0.815 ms
2 xe-2-0-0.car04.dllstx2.networklayer.com (67.18.7.93) 0.415 ms 0.428 ms 0.426 ms
3 po102.dsr01.dllstx2.networklayer.com (70.87.254.81) 0.756 ms 0.764 ms 0.963 ms
4 po21.dsr01.dllstx3.networklayer.com (70.87.255.65) 0.702 ms 0.759 ms 1.103 ms
5 ae16.bbr02.eq01.dal03.networklayer.com (173.192.18.228) 0.438 ms 0.446 ms 0.441 ms
6 ae7.bbr01.eq01.dal03.networklayer.com (173.192.18.208) 1.271 ms 0.709 ms 0.612 ms
7 ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 28.374 ms 28.431 ms 28.411 ms
8 ae7.bbr02.cs01.lax01.networklayer.com (173.192.18.167) 28.773 ms 28.747 ms 28.727 ms
9 * * *
10 ae0.bbr01.eq01.tok01.networklayer.com (50.97.18.161) 137.992 ms 138.001 ms 137.967 ms
11 ae7.bbr02.eq01.tok01.networklayer.com (50.97.18.163) 138.084 ms 138.068 ms 138.064 ms
12 ae0.bbr01.pn01.hkg01.networklayer.com (50.97.18.167) 181.952 ms 182.026 ms 181.990 ms
13 hutchcity21-10G.hkix.net (202.40.160.193) 180.121 ms 179.677 ms 179.631 ms
14 218.189.5.51 (218.189.5.51) 179.553 ms 179.521 ms 179.505 ms
15 d1-34-224-143-118-on-nets.com (118.143.224.34) 186.883 ms 186.829 ms 186.807 ms
16 * * *
17 omake.koitsu.org (208.79.90.130) 192.086 ms 192.973 ms 192.095 ms
The packet path for the above is:
Dallas (Linode then NetworkLayer)
-> LAX (NetworkLayer)
-> Tokyo (NetworkLayer)
-> Hong Kong (NetworkLayer then via HKIX)
-> Hong Kong (Hutchison Global Communications) (AS 9304)
-> ?
-> 208.79.90.130
Reminder: 208.79.90.130 is not physically in Hong Kong.
I haven't spent the time to look at bgplay.routeviews.org yet, but I'm
not sure it'll necessarily have insights into any of this.
I have reached out to ARP Networks to have them check things, but this
is an interesting situation I thought I'd mention as folks more familiar
with BGP than myself could probably assist with.
I can only speculate at this point, but I wonder if AS 9304 may have
announced some kind of preferencing that certain things have picked up
(but never withdrew/changed?). There are lots of explanations though,
so that's purely hearsay on my part.
--
| Jeremy Chadwick jdc at koitsu.org |
| UNIX Systems Administrator http://jdc.koitsu.org/ |
| Making life hard for others since 1977. PGP 4BD6C0CB |
More information about the Outages
mailing list