[outages] LAX routing anomaly: NetworkLayer? CoreSite? Hong Kong ISPs?
Jeremy Chadwick
jdc at koitsu.org
Mon Sep 30 13:33:43 EDT 2013
Follow-up:
I got lots of off-list responses about this, all of which were very
informative. Thanks everyone.
Wanted to report that after CoreSite did some maintenance this morning,
things are back in order:
src IP: 208.79.90.130
dst IP: 67.18.187.25
=== Mon Sep 30 02:56:00 PDT 2013 (1380534960)
Start: Mon Sep 30 02:56:00 2013
HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst
1.|-- 208.79.90.129 0.0% 40 40 0.9 5.6 0.8 128.4
2.|-- 208.79.88.135 0.0% 40 40 7.9 1.0 0.5 9.7
3.|-- 129.250.198.185 0.0% 40 40 0.7 0.8 0.7 1.2
4.|-- 129.250.5.69 0.0% 40 40 0.5 3.9 0.5 28.7
5.|-- 129.250.6.11 0.0% 40 40 10.3 13.1 9.6 35.4
6.|-- 129.250.5.53 0.0% 40 40 11.1 10.7 10.1 11.6
7.|-- 128.241.219.234 0.0% 40 40 162.8 165.4 161.7 206.4
8.|-- 173.192.18.151 0.0% 40 40 172.3 172.0 171.3 174.8
9.|-- 173.192.18.166 10.0% 40 36 156.7 156.6 155.6 160.6
10.|-- 173.192.18.140 10.0% 40 36 199.4 201.2 198.4 246.2
11.|-- 173.192.18.225 0.0% 40 40 184.1 185.5 182.9 222.8
12.|-- 70.87.255.66 0.0% 40 40 192.2 192.5 191.7 203.6
13.|-- 70.87.254.74 0.0% 40 40 193.0 194.8 192.0 229.3
14.|-- 67.18.7.90 2.5% 40 39 197.8 197.5 196.8 198.6
15.|-- 67.18.187.25 2.5% 40 39 200.5 201.4 200.5 204.1
=== END
=== Mon Sep 30 02:57:00 PDT 2013 (1380535020)
Start: Mon Sep 30 02:57:00 2013
HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst
1.|-- 208.79.90.129 0.0% 40 40 25.9 42.7 0.9 412.7
2.|-- 208.79.88.135 0.0% 40 40 0.5 77.9 0.4 163.2
3.|-- 129.250.198.185 5.0% 40 38 63.0 98.5 0.7 215.0
4.|-- 129.250.5.69 0.0% 40 40 29.1 95.9 0.6 261.0
5.|-- 129.250.6.11 0.0% 40 40 31.4 100.9 9.6 184.4
6.|-- 129.250.5.53 0.0% 40 40 28.5 100.0 10.1 224.1
7.|-- 128.241.219.234 0.0% 40 40 28.5 170.4 28.5 192.8
8.|-- 173.192.18.151 0.0% 40 40 31.5 175.6 31.4 198.7
9.|-- 173.192.18.166 2.5% 40 39 31.4 170.4 31.4 218.0
10.|-- 173.192.18.140 2.5% 40 39 31.4 188.5 31.4 244.7
11.|-- 173.192.18.225 0.0% 40 40 31.4 180.8 31.4 193.2
12.|-- 70.87.255.66 0.0% 40 40 31.4 184.5 31.4 193.1
13.|-- 70.87.254.74 0.0% 40 40 31.5 184.9 31.4 198.7
14.|-- 67.18.7.90 0.0% 40 40 31.4 187.0 31.4 199.0
15.|-- 67.18.187.25 0.0% 40 40 31.4 188.6 31.4 202.3
=== END
=== Mon Sep 30 02:58:00 PDT 2013 (1380535080)
Start: Mon Sep 30 02:58:00 2013
HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst
1.|-- 208.79.90.129 0.0% 40 40 9.3 16.4 0.9 177.8
2.|-- 206.223.143.131 0.0% 40 40 0.5 0.9 0.5 10.9
3.|-- 173.192.18.140 12.5% 40 35 28.1 28.6 28.1 34.2
4.|-- 173.192.18.225 0.0% 40 40 29.0 29.0 28.7 29.8
5.|-- 70.87.255.66 0.0% 40 40 31.5 31.5 31.3 32.2
6.|-- 70.87.254.74 0.0% 40 40 28.6 34.6 28.5 110.6
7.|-- 67.18.7.90 0.0% 40 40 28.6 28.6 28.4 29.0
8.|-- 67.18.187.25 0.0% 40 40 31.4 31.5 31.4 32.2
=== END
And the return path:
src IP: 67.18.187.25
dst IP: 208.79.90.130
traceroute to omake.koitsu.org (208.79.90.130), 30 hops max, 60 byte packets
1 router2-dal.linode.com (67.18.7.162) 0.455 ms 0.601 ms 0.788 ms
2 xe-2-0-0.car04.dllstx2.networklayer.com (67.18.7.93) 0.293 ms 0.313 ms 0.317 ms
3 po102.dsr01.dllstx2.networklayer.com (70.87.254.81) 0.562 ms 0.686 ms 0.760 ms
4 po21.dsr01.dllstx3.networklayer.com (70.87.255.65) 0.654 ms 0.731 ms 0.873 ms
5 ae16.bbr02.eq01.dal03.networklayer.com (173.192.18.228) 0.334 ms 0.334 ms 0.347 ms
6 ae7.bbr01.eq01.dal03.networklayer.com (173.192.18.208) 0.486 ms 0.578 ms 0.481 ms
7 ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 33.662 ms 33.719 ms 33.698 ms
8 any2-ix.la.arpnetworks.com (206.223.143.166) 34.656 ms 34.998 ms 35.247 ms
9 omake.koitsu.org (208.79.90.130) 31.298 ms 31.299 ms 31.285 ms
--
| Jeremy Chadwick jdc at koitsu.org |
| UNIX Systems Administrator http://jdc.koitsu.org/ |
| Making life hard for others since 1977. PGP 4BD6C0CB |
On Sun, Sep 29, 2013 at 07:31:21PM -0700, Jeremy Chadwick wrote:
> I just came across this on NANOG while troubleshooting something:
>
> http://mailman.nanog.org/pipermail/nanog/2013-September/061332.html
>
> If the issue described there is what I'm seeing, then it looks like
> some part of the 'net in the LA area is very very broken -- and worse,
> *has* been broken since roughly Sep 27 21:16:00 PDT (UTC-7).
>
> What I saw (and am still seeing):
>
> src IP: 208.79.90.130 (Southern CA, AS 25795 (I think))
> dst IP: 67.18.187.25 (Texas, AS 21844 (I think))
>
> === Fri Sep 27 21:14:00 PDT 2013 (1380341640)
> Start: Fri Sep 27 21:14:00 2013
> HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst
> 1.|-- 208.79.90.129 0.0% 40 40 3.7 11.0 0.9 202.9
> 2.|-- 206.223.143.131 0.0% 40 40 0.6 0.8 0.5 8.5
> 3.|-- 173.192.18.140 2.5% 40 39 28.4 30.1 28.2 86.8
> 4.|-- 173.192.18.225 0.0% 40 40 29.0 29.2 28.7 40.9
> 5.|-- 70.87.255.66 0.0% 40 40 31.6 31.5 31.3 31.9
> 6.|-- 70.87.254.74 0.0% 40 40 29.0 31.5 28.6 60.3
> 7.|-- 67.18.7.90 0.0% 40 40 28.7 28.6 28.4 29.0
> 8.|-- 67.18.187.25 0.0% 40 40 31.9 31.5 31.3 31.9
> === END
>
> === Fri Sep 27 21:15:00 PDT 2013 (1380341700)
> Start: Fri Sep 27 21:15:00 2013
> HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst
> 1.|-- 208.79.90.129 0.0% 40 40 2.2 51.1 0.9 245.7
> 2.|-- 206.223.143.131 2.5% 40 39 0.9 12.3 0.5 183.2
> 3.|-- 173.192.18.140 5.0% 40 38 0.8 29.4 0.7 284.1
> 4.|-- 173.192.18.225 0.0% 40 40 0.8 17.1 0.5 177.8
> 5.|-- 70.87.255.66 30.0% 40 28 11.2 39.1 3.4 199.4
> 6.|-- 70.87.254.74 0.0% 40 40 11.0 21.1 0.5 121.5
> 7.|-- 67.18.7.90 0.0% 40 40 163.8 122.6 4.6 347.0
> 8.|-- 67.18.187.25 22.5% 40 31 171.6 81.4 0.5 173.8
> === END
>
> === Fri Sep 27 21:16:00 PDT 2013 (1380341760)
> Start: Fri Sep 27 21:16:00 2013
> HOST: omake.koitsu.org Loss% Snt Rcv Last Avg Best Wrst
> 1.|-- 208.79.90.129 0.0% 40 40 2.9 7.4 0.9 95.8
> 2.|-- 208.79.88.135 0.0% 40 40 0.6 15.8 0.4 182.3
> 3.|-- 129.250.198.185 0.0% 40 40 0.7 0.8 0.7 1.2
> 4.|-- 129.250.5.69 0.0% 40 40 0.7 5.1 0.6 34.1
> 5.|-- 129.250.6.11 45.0% 40 22 17.8 14.6 9.6 47.1
> 6.|-- 129.250.5.53 0.0% 40 40 11.6 10.8 10.0 12.2
> 7.|-- 128.241.219.234 0.0% 40 40 162.7 163.1 161.7 178.3
> 8.|-- 173.192.18.151 2.5% 40 39 172.3 172.3 171.2 176.3
> 9.|-- 173.192.18.166 12.5% 40 35 157.2 156.3 155.6 157.9
> 10.|-- 173.192.18.140 7.5% 40 37 198.5 201.1 198.4 234.7
> 11.|-- 173.192.18.225 0.0% 40 40 183.0 190.1 182.9 355.2
> 12.|-- 70.87.255.66 0.0% 40 40 183.0 183.8 183.0 185.9
> 13.|-- 70.87.254.74 0.0% 40 40 193.6 195.8 192.0 284.1
> 14.|-- 67.18.7.90 2.5% 40 39 197.0 197.4 196.9 198.8
> 15.|-- 67.18.187.25 2.5% 40 39 192.8 192.8 192.2 194.0
> === END
>
> FYI: 206.223.143.131 resolves to te2-6.bbr01.cs01.lax01.networklayer.com.any2ix.coresite.com.
>
> ARIN states 206.223.143.0/24 is CoreSite.
>
> The packet path for the above is:
>
> LAX (ARP Networks)
> -> SFO (NTT/Verio)
> -> San Jose (NTT/Verio)
> -> LAX (NetworkLayer)
> -> Dallas (NetworkLayer and Linode)
> -> 67.18.187.25
>
> And now for the amusing part -- the return path:
>
> src IP: 67.18.187.25 (Texas, AS 21844 (I think))
> dst IP: 208.79.90.130 (Southern CA, AS 25795 (I think))
>
> traceroute to omake.koitsu.org (208.79.90.130), 30 hops max, 60 byte packets
> 1 router2-dal.linode.com (67.18.7.162) 0.604 ms 0.812 ms 0.815 ms
> 2 xe-2-0-0.car04.dllstx2.networklayer.com (67.18.7.93) 0.415 ms 0.428 ms 0.426 ms
> 3 po102.dsr01.dllstx2.networklayer.com (70.87.254.81) 0.756 ms 0.764 ms 0.963 ms
> 4 po21.dsr01.dllstx3.networklayer.com (70.87.255.65) 0.702 ms 0.759 ms 1.103 ms
> 5 ae16.bbr02.eq01.dal03.networklayer.com (173.192.18.228) 0.438 ms 0.446 ms 0.441 ms
> 6 ae7.bbr01.eq01.dal03.networklayer.com (173.192.18.208) 1.271 ms 0.709 ms 0.612 ms
> 7 ae0.bbr01.cs01.lax01.networklayer.com (173.192.18.141) 28.374 ms 28.431 ms 28.411 ms
> 8 ae7.bbr02.cs01.lax01.networklayer.com (173.192.18.167) 28.773 ms 28.747 ms 28.727 ms
> 9 * * *
> 10 ae0.bbr01.eq01.tok01.networklayer.com (50.97.18.161) 137.992 ms 138.001 ms 137.967 ms
> 11 ae7.bbr02.eq01.tok01.networklayer.com (50.97.18.163) 138.084 ms 138.068 ms 138.064 ms
> 12 ae0.bbr01.pn01.hkg01.networklayer.com (50.97.18.167) 181.952 ms 182.026 ms 181.990 ms
> 13 hutchcity21-10G.hkix.net (202.40.160.193) 180.121 ms 179.677 ms 179.631 ms
> 14 218.189.5.51 (218.189.5.51) 179.553 ms 179.521 ms 179.505 ms
> 15 d1-34-224-143-118-on-nets.com (118.143.224.34) 186.883 ms 186.829 ms 186.807 ms
> 16 * * *
> 17 omake.koitsu.org (208.79.90.130) 192.086 ms 192.973 ms 192.095 ms
>
> The packet path for the above is:
>
> Dallas (Linode then NetworkLayer)
> -> LAX (NetworkLayer)
> -> Tokyo (NetworkLayer)
> -> Hong Kong (NetworkLayer then via HKIX)
> -> Hong Kong (Hutchison Global Communications) (AS 9304)
> -> ?
> -> 208.79.90.130
>
> Reminder: 208.79.90.130 is not physically in Hong Kong.
>
> I haven't spent the time to look at bgplay.routeviews.org yet, but I'm
> not sure it'll necessarily have insights into any of this.
>
> I have reached out to ARP Networks to have them check things, but this
> is an interesting situation I thought I'd mention as folks more familiar
> with BGP than myself could probably assist with.
>
> I can only speculate at this point, but I wonder if AS 9304 may have
> announced some kind of preferencing that certain things have picked up
> (but never withdrew/changed?). There are lots of explanations though,
> so that's purely hearsay on my part.
>
> --
> | Jeremy Chadwick jdc at koitsu.org |
> | UNIX Systems Administrator http://jdc.koitsu.org/ |
> | Making life hard for others since 1977. PGP 4BD6C0CB |
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
More information about the Outages
mailing list