[outages] HE.NET Columbus Latency & Packet Loss
Adam
adamkuj at gmail.com
Wed Jan 23 23:22:29 EST 2019
Hi Glenn,
I saw the same thing this morning and had to shift all our traffic away.
They emailed shortly after 5:30pm to let me know it was resolved. Their
100GigE wave from Columbus to Chicago was down.
On an unrelated note, are you interested in joining Ohio IX? We (Amplex
Internet) just joined, and I'm working with them to onboard some of the
larger networks (Amazon, Google, Netflix).
More info:
http://ohioix.net/
https://www.peeringdb.com/ix/869
-Adam
On Wed, Jan 23, 2019 at 1:50 PM Glenn Kelley via Outages <
outages at outages.org> wrote:
> Just confirmed with the staff @ HE.net they are experiencing packet loss
> and latency from the Columbus Coligix POP.
>
> I have a feeling it is more since we have customers in Indianapolis as
> well as Chicago Markets now reporting issues.
>
>
> #tool traceroute address=hulu.com use-dns=yes
> # ADDRESS LOSS SENT LAST AVG BEST
> WORST STD-DEV STATUS
> 1 peering_fb.horizon-ce.10ge-1-... 0% 492 4.4ms 4.4 4.3
> 5.3 0.1
> 2 edge.hurricianelectric-net.10... 0% 492 4.4ms 4.5 4.4
> 5.7 0.1
> 3 v212.core1.cmh1.he.net 0% 492 4.7ms 5.3 4.6
> 32.9 2.8
> 4 10ge4-1.core1.ind1.he.net 0% 492 11.5ms 11.4 11.1
> 14.5 0.4
> 5 100ge16-2.core1.chi1.he.net 7.5% 492 43.7ms 41.9 37.4
> 61.9 40.8
> 6 chi-b21-link.telia.net 8.7% 492 42.1ms 42.1 38.9
> 49.6 41.3
> 7 ix-ae-24-0.tcore1.ct8-chicago... 6.9% 492 41.7ms 42.1 37.9
> 75.3 40.6
> 8 if-ae-26-2.tcore2.nto-new-yor... 0.6% 492 118.5ms 119.2 118.2
> 156.1 119.2 <MPLS:L=334955,E=0>
> 9 if-ae-32-2.tcore2.ldn-london.... 0% 492 109.9ms 110.7 109.8
> 153.3 110.7
> 10 80.231.20.42 0% 492 95.2ms 95.7 95.1
> 122.4 95.1
> 11 a23-214-189-19.deploy.static.... 0% 492 93.4ms 93.5 93.2
> 99.4 93.2
> -- [Q quit|D dump|C-z pause]
>
>
>
> tool traceroute address=amazon.com use-dns=yes
> # ADDRESS LOSS SENT LAST AVG BEST
> WORST STD-DEV STATUS
> 1 peering_fb.horizon-ce.10ge-1-... 0% 6 4.3ms 4.5 4.3
> 4.5 0.1
> 2 edge.hurricianelectric-net.10... 0% 6 4.6ms 4.5 4.4
> 4.6 0.1
> 3 v212.core1.cmh1.he.net 0% 6 4.8ms 4.7 4.7
> 4.8 0
> 4 10ge4-1.core1.ind1.he.net 0% 6 11.3ms 11.5 11.3
> 12 0.2
> 5 100ge16-2.core1.chi1.he.net 16.. 6 43.3ms 42 40.1
> 43.3 1.2
> 6 100ge2-2.core2.chi1.he.net 33.. 6 42.2ms 42.7 40
> 44.7 1.8
> 7 equinix01-chi2.amazon.com 0% 6 41.8ms 42.8 40.3
> 45 1.5
> 8 52.95.62.68 0% 6 27.5ms 27.8 27.5
> 28.4 0.3 <MPLS:L=489056,E=0 L=2807,E=0,T=1>
> 9 52.95.62.81 0% 6 29.6ms 27.7 27.1
> 29.6 0.9 <MPLS:L=2807,E=0>
> 10 100% 6 timeout
>
> 11 54.239.44.77 0% 6 32.4ms 29.2 28.5
> 32.4 1.5 <MPLS:L=516692,E=0>
> 12 100% 6 timeout
>
> 13 100% 6 timeout
>
> 14 100% 6 timeout
>
> 15 100% 5 timeout
>
>
> (this network on the Amazon Cloud blocks ICMP - so the results before the
> timeouts are what we are actually looking @ since it shows drops on the HE
> backbone)
>
>
>
> tool traceroute address=isup.me use-dns=yes
> # ADDRESS LOSS SENT LAST AVG BEST
> WORST STD-DEV STATUS
> 1 peering_fb.horizon-ce.10ge-1-... 0% 6 4.5ms 4.5 4.4
> 4.6 0.1
> 2 edge.hurricianelectric-net.10... 0% 6 4.6ms 4.5 4.4
> 4.6 0.1
> 3 v212.core1.cmh1.he.net 0% 6 4.7ms 7.1 4.6
> 19.1 5.4
> 4 10ge4-1.core1.ind1.he.net 0% 6 11.2ms 11.2 11.2
> 11.3 0
> 5 100ge16-2.core1.chi1.he.net 0% 6 40.2ms 41.9 40.2
> 43.9 1.2
> 6 100ge2-2.core2.chi1.he.net 0% 6 41.1ms 41.6 39.6
> 44.4 1.6
> 7 100ge14-2.core1.msp1.he.net 0% 6 48ms 50.7 48
> 52.5 1.7
> 8 100ge10-1.core1.sea1.he.net 16.. 6 93.3ms 85 81.3
> 93.3 4.3
> 9 six01-sea4.amazon.com 0% 6 83.7ms 83 81.2
> 84.1 1.2
> 10 52.95.52.86 16.. 6 81.7ms 83.6 81.2
> 85.7 1.8 <MPLS:L=656224,E=0>
> 11 52.95.52.97 0% 6 84.8ms 83.4 80.2
> 84.8 1.5
> 12 100% 6 timeout
>
> 13 54.239.45.119 0% 6 87.1ms 89.3 87.1
> 91.1 1.4 <MPLS:L=361254,E=0>
> 14 100% 6 timeout
>
> 15 52.93.12.188 40% 5 timeout 98.8 90.1
> 114.6 11.2 <MPLS:L=347944,E=0>
> 16 52.93.12.209 0% 5 89.8ms 89.1 85.9
> 91.1 1.9
> 17 52.93.240.53 0% 5 88.8ms 89.6 88.8
> 91.1 0.9
> 18 100% 5 timeout
>
> 19 100% 5 timeout
>
>
> (again end site blocks ICMP we expect those timeouts - the latency in the
> HE.network is what is of concern)
>
>
> tool traceroute address=gmail.com use-dns=yes
> # ADDRESS LOSS SENT LAST AVG BEST
> WORST STD-DEV STATUS
> 1 peering_fb.horizon-ce.10ge-1-... 0% 25 4.5ms 4.4 4.4
> 4.6 0.1
> 2 edge.hurricianelectric-net.10... 0% 25 4.5ms 4.5 4.4
> 4.6 0.1
> 3 v212.core1.cmh1.he.net 0% 25 4.8ms 4.8 4.7
> 4.9 0.1
> 4 10ge4-1.core1.ind1.he.net 0% 25 11.1ms 11.2 11.1
> 11.8 0.1
> 5 100ge16-2.core1.chi1.he.net 8% 25 41ms 41.2 38.8
> 43.3 1.4
> 6 100ge2-2.core2.chi1.he.net 16% 25 39ms 41.3 39
> 44.2 1.3
> 7 eqix-ch-100g.google.com 8% 25 41ms 42.2 38.9
> 53 2.7
> 8 108.170.243.225 8% 25 39.6ms 43 39.6
> 47.7 1.8
> 9 209.85.249.73 0% 25 15.2ms 15.2 15.1
> 15.4 0.1
> 10 ord31s22-in-f229.1e100.net 4% 25 40.9ms 41.5 39.1
> 43.3 1.2
>
>
> tool traceroute address=outlook.com use-dns=yes
> # ADDRESS LOSS SENT LAST AVG BEST
> WORST STD-DEV STATUS
>
> 1 peering_fb.horizon-ce.10ge-1-... 0% 10 4.4ms 4.5 4.3
> 4.5 0.1
>
> 2 edge.hurricianelectric-net.10... 0% 10 4.6ms 4.5 4.4
> 4.6 0.1
>
> 3 v212.core1.cmh1.he.net 0% 10 5.2ms 4.7 4.6
> 5.2 0.2
>
> 4 10ge4-1.core1.ind1.he.net 0% 10 11.3ms 11.7 11.2
> 15.5 1.3
>
> 5 100ge16-2.core1.chi1.he.net 0% 10 39ms 40.8 39
> 43.7 1.4
>
> 6 100ge2-2.core2.chi1.he.net 10% 10 41.9ms 41.3 39.9
> 42.3 0.8
>
> 7 206.108.115.47 10% 10 40ms 41.4 40
> 43.6 1.1
>
> 8 ae4-0.ch1-96c-2b.ntwk.msn.net 0% 10 42.3ms 46.2 41.1
> 62.3 6.3 <MPLS:L=16841,E=0>
>
> 9 be-64-0.ibr01.ch1.ntwk.msn.net 0% 10 41.5ms 41.8 40.8
> 43 0.7
>
> 10 be-5-0.ibr01.ch2.ntwk.msn.net 0% 10 40.6ms 40.7 40.3
> 41 0.2
>
> 11 be-6-0.ibr01.cnr02.dsm05.ntwk... 0% 10 40.6ms 41.7 40.6
> 43.1 0.8
>
> 12 be-3-0.ibr01.cnr03.dsm05.ntwk... 0% 10 40.9ms 41.5 40.9
> 42 0.4
>
> 13 be-5-0.ibr02.den07.ntwk.msn.net 0% 10 41.7ms 41.5 40.7
> 42.4 0.5
>
> 14 be-1-0.ibr01.den07.ntwk.msn.net 0% 10 41.9ms 41.5 40.8
> 42.2 0.5
>
> 15 be-5-0.ibr02.cys04.ntwk.msn.net 10% 10 40.4ms 40.8 40.4
> 42 0.5
>
> 16 ae162-0.icr02.cys04.ntwk.msn.net 0% 10 40.3ms 40.2 39.9
> 41 0.3
>
> 17 100% 10 timeout
>
>
> 18 100% 10 timeout
>
>
>
>
>
> tool traceroute address=pornhub.com use-dns=yes
> # ADDRESS LOSS SENT LAST AVG BEST
> WORST STD-DEV STATUS
>
> 1 peering_fb.horizon-ce.10ge-1-... 0% 42 4.5ms 4.5 4.4
> 4.6 0.1
>
> 2 edge.hurricianelectric-net.10... 0% 42 4.4ms 4.5 4.4
> 4.7 0.1
>
> 3 v212.core1.cmh1.he.net 0% 42 4.8ms 4.9 4.7
> 9.6 0.8
>
> 4 10ge4-1.core1.ind1.he.net 0% 42 11.4ms 11.3 11.1
> 11.6 0.1
>
> 5 100ge16-2.core1.chi1.he.net 2.4% 42 41ms 41.9 38.6
> 57.3 3
>
> 6 chi-b21-link.telia.net 7.1% 42 41.8ms 42 39.5
> 48.4 1.6
>
> 7 haproxy-ic-332713-chi-b21.c.t... 14.. 42 46.8ms 44 39.4
> 64.8 4.8
>
> 8 216.18.168.16 11.. 42 42ms 41.7 38.5
> 45.6 1.6
>
>
>
>
> tool traceroute address=microsoft.com use-dns=yes
> # ADDRESS LOSS SENT LAST AVG BEST
> WORST STD-DEV STATUS
>
> 1 peering_fb.horizon-ce.10ge-1-... 0% 5 4.4ms 4.5 4.4
> 4.6 0.1
>
> 2 edge.hurricianelectric-net.10... 0% 5 4.5ms 4.5 4.5
> 4.6 0
>
> 3 v212.core1.cmh1.he.net 0% 5 4.6ms 4.7 4.6
> 4.8 0.1
>
> 4 10ge4-1.core1.ind1.he.net 0% 5 11.3ms 11.3 11.2
> 11.4 0.1
>
> 5 100ge16-2.core1.chi1.he.net 20% 5 42.5ms 43.1 42.2
> 44.3 0.8
>
> 6 100ge2-2.core2.chi1.he.net 0% 5 42.7ms 44.3 39.6
> 52 4.4
>
> 7 chi-8075.msn.net 20% 5 40.9ms 42.9 40.9
> 46.5 2.1
>
> 8 ae12-0.ch1-96c-2a.ntwk.msn.net 20% 5 42.5ms 44.3 41.8
> 46.7 2.1 <MPLS:L=16835,E=0>
>
> 9 be-63-0.ibr01.ch1.ntwk.msn.net 0% 5 27.1ms 28.1 27.1
> 29.1 0.8
>
> 10 be-5-0.ibr01.ch2.ntwk.msn.net 0% 5 27.1ms 27.7 27.1
> 28.9 0.6
>
> 11 be-3-0.ibr01.was02.ntwk.msn.net 0% 5 28.9ms 28.4 27.5
> 29.3 0.6
>
> 12 be-5-0.ibr03.bl7.ntwk.msn.net 0% 5 27.3ms 27.5 27.3
> 27.8 0.2
>
> 13 ae142-0.icr02.bl7.ntwk.msn.net 0% 5 31.6ms 30.4 27
> 37.9 4.1
>
> 14 100% 5 timeout
>
>
> 15 100% 5 timeout
>
>
> 16 100% 5 timeout
>
>
> 17 100% 4 timeout
>
>
> 18 100% 4 timeout
>
>
>
> We ran most of these for minutes on end - just captured the snapshots
> briefly however
>
>
>
> --
> Glenn Kelley
> www.Connectivity.Engineer
> EMT-P, CCNA, MTCINE, MTCRE,CFOS/DC, CFOS/A
>
>
> -------------------------------------------------------------------------------------------------
> Connectivity.Engineer is a family of companies including
> Hop Off A Cloud Networks, Typo3USA PasturePress, White Lab Studios & the
> CARM Network
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20190123/3823d40c/attachment.htm>
More information about the Outages
mailing list