[Outages-discussion] [outages] OVH IPv6 100% Package Loss in BHS (Canada DC)
Jim Popovitch
jimpop at domainmail.org
Mon Sep 7 14:28:54 EDT 2020
On Mon, 2020-09-07 at 14:18 -0400, Jim Popovitch wrote:
> On Mon, 2020-09-07 at 20:15 +0300, Vlad Dascalu via Outages wrote:
> > Hey
> >
> > Does anyone have a proper contact point for OVH NOC (besides noc at ovh.net ?)
> >
> > Traceroute is showing 100% package loss on an IPv6 path in their AS, despite being routable from other ASes. I've reported it for hours but have no reaction from them. Is it normal?
> >
> > ( Should we, as an Internet community, make a plan of attack regarding accountability on responsiveness for large ISP providers in the light of the recent Level3 incident? )
> >
> > Anyway, below you can find the problematic traceroute.
> >
> > # wget http://[2607:5300:60:6195::1]
> > --2020-09-07 05:39:39-- http://[2607:5300:60:6195::1]/
> > Connecting to [2607:5300:60:6195::1]:80... failed: Connection timed out.
> >
> > # traceroute 2607:5300:60:6195::1
> > traceroute to 2607:5300:60:6195::1 (2607:5300:60:6195::1), 30 hops max, 80 byte packets
> > 1 * * *
> > 2 xe-0-0-25-1.a01.atlnga05.us.bb.gin.ntt.net (2001:418:0:5000::622) 1.012 ms 1.012 ms 1.017 ms
> > 3 ae-6.r23.atlnga05.us.bb.gin.ntt.net (2001:418:0:2000::22a) 0.635 ms ae-16.r22.atlnga05.us.bb.gin.ntt.net (2001:418:0:2000::20e) 1.004 ms ae-6.r23.atlnga05.us.bb.gin.ntt.net (2001:418:0:2000::22a) 0.646 ms
> > 4 ae-9.r04.atlnga05.us.bb.gin.ntt.net (2001:418:0:2000::14d) 1.077 ms 1.079 ms 1.075 ms
> > 5 2001:418:0:4000::a2 (2001:418:0:4000::a2) 1.052 ms * 1.036 ms
> > 6 rest-bb1-v6.telia.net (2001:2000:3019:73::1) 16.727 ms ash-bb2-v6.telia.net (2001:2000:3019:74::1) 13.431 ms 13.480 ms
> > 7 * * *
> > 8 be100-1007.nwk-5-a9.nj.us (2607:5300::2) 17.916 ms * *
> > 9 be102.bhs-g2-nc5.qc.ca (2607:5300::1ce) 27.694 ms 25.405 ms 29.696 ms
> > 10 vl100.bhs-d2-a75.qc.ca (2607:5300::1cd) 27.329 ms 27.455 ms 24.885 ms
> > 11 2001:41d0:0:50::6:80b (2001:41d0:0:50::6:80b) 24.216 ms 24.391 ms 24.381 ms
> > 12 * * *
> > 13 * * *
> > 14 * * *
> > 15 * * *
> > 16 * * *
> > 17 * * *
> > 18 * * *
> > 19 * * *
> > 20 * * *
> > 21 * * *
> > 22 * * *
> > 23 * * *
> > 24 * * *
> > 25 * * *
> > 26 * * *
> > 27 * * *
> > 28 * * *
> > 29 * * *
> > 30 * * *
> >
> > If it helps, I tested connectivity through 3 other continents and they work ok. Seems BGP resolves it to the right ASN but there is a routing dead-end inside the BHS router config.
> >
> > Traceroute working from other locations, if it helps:
> >
> > Germany (Hetzner):
> > traceroute to ca1.beetux.com (2607:5300:60:6195::1), 30 hops max, 80 byte packets
> > 1 ex9k1.dc7.fsn1.hetzner.com (2a01:4f8::a:16:a) 0.489 ms 0.478 ms 0.471 ms
> > 2 core23.fsn1.hetzner.com (2a01:4f8:0:3::271) 0.464 ms core24.fsn1.hetzner.com (2a01:4f8:0:3::275) 0.434 ms core23.fsn1.hetzner.com (2a01:4f8:0:3::271) 0.447 ms
> > 3 core5.fra.hetzner.com (2a01:4f8:0:3::33a) 16.099 ms core0.fra.hetzner.com (2a01:4f8:0:3::365) 5.107 ms core5.fra.hetzner.com (2a01:4f8:0:3::33a) 16.090 ms
> > 4 core9.fra.hetzner.com (2a01:4f8:0:3::2fa) 5.438 ms 5.433 ms core8.fra.hetzner.com (2a01:4f8:0:3::33d) 5.409 ms
> > 5 * * *
> > 6 2001:41d0::257b (2001:41d0::257b) 6.192 ms * 2001:41d0::2579 (2001:41d0::2579) 5.766 ms
> > 7 2001:41d0:aaaa:100::3 (2001:41d0:aaaa:100::3) 22.947 ms 16.798 ms 23.640 ms
> > 8 * * 2001:41d0:aaaa:100::2 (2001:41d0:aaaa:100::2) 6.062 ms
> > 9 * * *
> > 10 * * *
> > 11 * * *
> > 12 be102.bhs-g1-nc5.qc.ca (2607:5300::1c6) 95.367 ms 96.431 ms be102.bhs-g2-nc5.qc.ca (2607:5300::1ce) 98.105 ms
> > 13 vl100.bhs-d1-a75.qc.ca (2607:5300::1c3) 94.740 ms vl100.bhs-d2-a75.qc.ca (2607:5300::1cd) 95.564 ms 93.632 ms
> > 14 2001:41d0:0:50::6:809 (2001:41d0:0:50::6:809) 94.240 ms 2001:41d0:0:50::6:80d (2001:41d0:0:50::6:80d) 91.366 ms 2001:41d0:0:50::6:80b (2001:41d0:0:50::6:80b) 94.574 ms
> > 15 * * *
> > 16 ca1.beetux.com (2607:5300:60:6195::1) 91.282 ms !X 92.104 ms !X 92.496 ms !X
> >
> > Brazil:
> > traceroute to ca1.beetux.com (2607:5300:60:6195::1), 30 hops max, 80 byte packet s
> > 1 hsrp.gcore.lu (2a03:90c0:186::3) 0.704 ms 0.457 ms 0.669 ms
> > 2 2a03:90c0:180:1:253::226 (2a03:90c0:180:1:253::226) 0.695 ms 0.668 ms 0.6 44 ms
> > 3 g-core-labs.sanpaolo7.spa.seabone.net (2001:41a8:5200:2::1e1) 0.734 ms 2001 :1498:1:bf1::1 (2001:1498:1:bf1::1) 0.723 ms 0.701 ms
> > 4 mia15-loop0-v6.mia.seabone.net (2001:41a8:4020::f) 109.816 ms 110.905 ms 2 001:1498:1::100:138 (2001:1498:1::100:138) 134.126 ms
> > 5 2001:1498:1::32:113 (2001:1498:1::32:113) 110.835 ms * 2001:1498:1::32:146 (2001:1498:1::32:146) 131.040 ms
> > 6 * * *
> > 7 * be102.bhs-g1-nc5.qc.ca (2607:5300::1c6) 137.615 ms *
> > 8 vl100.bhs-d1-a75.qc.ca (2607:5300::1c3) 137.997 ms * be102.bhs-g1-nc5.qc.ca (2607:5300::1c6) 133.250 ms
> > 9 vl100.bhs-d1-a75.qc.ca (2607:5300::1c3) 132.065 ms 2001:41d0:0:50::6:80d (2 001:41d0:0:50::6:80d) 130.865 ms vl100.bhs-d1-a75.qc.ca (2607:5300::1c3) 132.1 55 ms
> > 10 2001:41d0:0:50::6:80f (2001:41d0:0:50::6:80f) 132.028 ms * *
> > 11 ca1.beetux.com (2607:5300:60:6195::1) 121.118 ms !X 132.094 ms !X 148.398 ms !X
> >
> > Singapore:
> > traceroute to ca1.beetux.com (2607:5300:60:6195::1), 30 hops max, 80 byte packets
> > 1 2400:6180:0:d1:ffff:ffff:ffff:fff2 (2400:6180:0:d1:ffff:ffff:ffff:fff2) 0.659 ms 0.638 ms 0.619 ms
> > 2 2604:a880:ffff:9::26a (2604:a880:ffff:9::26a) 0.960 ms 2604:a880:ffff:9::276 (2604:a880:ffff:9::276) 1.077 ms 1.057 ms
> > 3 2604:a880:ffff:9::265 (2604:a880:ffff:9::265) 0.865 ms 2604:a880:ffff:9::271 (2604:a880:ffff:9::271) 0.884 ms 2604:a880:ffff:9::263 (2604:a880:ffff:9::263) 0.835 ms
> > 4 2604:a880:ffff:9:0:1:0:6 (2604:a880:ffff:9:0:1:0:6) 1.148 ms 2604:a880:ffff:9:0:1:0:e (2604:a880:ffff:9:0:1:0:e) 1.135 ms 2604:a880:ffff:9:0:1:0:4 (2604:a880:ffff:9:0:1:0:4) 1.105 ms
> > 5 2403:e800:508:100::1 (2403:e800:508:100::1) 1.205 ms * *
> > 6 * * *
> > 7 * * *
> > 8 * * *
> > 9 * * *
> > 10 be100-104.pao-sv8-bb1-a9.ca.us (2607:5300::24) 168.764 ms 186.737 ms 186.765 ms
> > 11 be100-1371.chi-5-a9.il.us (2607:5300::af) 211.537 ms be100-1370.chi-1-a9.il.us (2607:5300::59) 222.876 ms 204.043 ms
> > 12 be103.bhs-g1-nc5.qc.ca (2607:5300::1c8) 408.785 ms 265.277 ms 230.064 ms
> > 13 vl100.bhs-d1-a75.qc.ca (2607:5300::1c3) 230.897 ms 230.898 ms 230.683 ms
> > 14 2001:41d0:0:50::6:809 (2001:41d0:0:50::6:809) 228.107 ms 246.649 ms 2001:41d0:0:50::6:80b (2001:41d0:0:50::6:80b) 250.099 ms
> > 15 * * *
> > 16 ca1.beetux.com (2607:5300:60:6195::1) 244.269 ms !X 230.678 ms !X 245.223 ms !X
>
> First, your post is not worthy of the outages at outages.org list. Please
> use outages-discussion at outages.org in the future. Real outages should
> go to outages at outages.org BUT understand that posting there will ring
> some peoples pagers/mobiles and other's inbox whistles. Please reserve
> outages at outages.org for real outages.
>
> Secondly, OVH has never had perfect IPv6 routing, and generally doesn't
> care what you or I have to say about their IPv6/routing choices.
>
> Now, you need to know that traceroute is an old tool that no longer is
> accurate for testing purposes. Use something like mtr instead. Here's
> an mtr to that IP from Atlanta:
>
> ~$ mtr -c 1 --report-wide --aslookup --show-ips 2607:5300:60:6195::1
> Start: 2020-09-07T14:10:18-0400
> HOST: work Loss% Snt Last Avg Best Wrst StDev
> 1. AS??? ap-ipv6-1494-udp.vpn (fd10:10:23::1) 0.0% 1 12.7 12.7 12.7 12.7 0.0
> 2. AS??? ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
> 3. AS2914 xe-0-0-25-1.a01.atlnga05.us.bb.gin.ntt.net (2001:418:0:5000::622) 0.0% 1 12.9 12.9 12.9 12.9 0.0
> 4. AS2914 ae-16.r22.atlnga05.us.bb.gin.ntt.net (2001:418:0:2000::20e) 0.0% 1 24.8 24.8 24.8 24.8 0.0
> 5. AS2914 ae-9.r04.atlnga05.us.bb.gin.ntt.net (2001:418:0:2000::14d) 0.0% 1 13.3 13.3 13.3 13.3 0.0
> 6. AS2914 2001:418:0:4000::a2 0.0% 1 12.7 12.7 12.7 12.7 0.0
> 7. AS1299 ash-bb2-v6.telia.net (2001:2000:3019:74::1) 0.0% 1 26.3 26.3 26.3 26.3 0.0
> 8. AS??? ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
> 9. AS16276 be100-1007.nwk-5-a9.nj.us (2607:5300::2) 0.0% 1 32.3 32.3 32.3 32.3 0.0
> 10. AS16276 be102.bhs-g2-nc5.qc.ca (2607:5300::1ce) 0.0% 1 38.7 38.7 38.7 38.7 0.0
> 11. AS16276 vl100.bhs-d2-a75.qc.ca (2607:5300::1cd) 0.0% 1 36.5 36.5 36.5 36.5 0.0
> 12. AS16276 2001:41d0:0:50::6:80f 0.0% 1 39.3 39.3 39.3 39.3 0.0
> 13. AS16276 2001:41d0:0:50::2:404d 0.0% 1 37.1 37.1 37.1 37.1 0.0
> 14. AS??? ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
>
> Here is a mtr from San Jose:
> Start: 2020-09-07T18:14:07+0000
> HOST: sjc Loss% Snt Last Avg Best Wrst StDev
> 1. AS??? ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
> 2. AS20473 2001:19f0:ac00:2300::1 0.0% 1 0.5 0.5 0.5 0.5 0.0
> 3. AS20473 2001:19f0:ac00::ada:15d 0.0% 1 1.1 1.1 1.1 1.1 0.0
> 4. AS??? ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
> 5. AS174 2001:550:2:3::b6:1 0.0% 1 1.1 1.1 1.1 1.1 0.0
> 6. AS??? ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
> 7. AS174 be3142.ccr41.sjc03.atlas.cogentco.com (2001:550:0:1000::9a36:1c2) 0.0% 1 2.1 2.1 2.1 2.1 0.0
> 8. AS16276 sjo-sv5-bb1-a9.ca.us (2607:5300::255e) 0.0% 1 1.9 1.9 1.9 1.9 0.0
> 9. AS16276 be100-1368.pao-sv8-bb1-a9.ca.us (2607:5300::26) 0.0% 1 22.1 22.1 22.1 22.1 0.0
> 10. AS16276 be100-1370.chi-1-a9.il.us (2607:5300::59) 0.0% 1 49.6 49.6 49.6 49.6 0.0
> 11. AS16276 be103.bhs-g1-nc5.qc.ca (2607:5300::1c8) 0.0% 1 72.9 72.9 72.9 72.9 0.0
> 12. AS16276 vl100.bhs-d1-a75.qc.ca (2607:5300::1c3) 0.0% 1 72.1 72.1 72.1 72.1 0.0
> 13. AS16276 2001:41d0:0:50::6:809 0.0% 1 71.8 71.8 71.8 71.8 0.0
> 14. AS16276 2001:41d0:0:50::2:4049 0.0% 1 73.5 73.5 73.5 73.5 0.0
> 15. AS16276 ca1.beetux.com (2607:5300:60:6195::1) 0.0% 1 71.9 71.9 71.9 71.9 0.0
>
> Those 2 mtr's show me that the drop is interestingly close to your host.
> What do you get mtr'ing from your OVH host to 2604:180::9b07:1d5b
> (RamNode's ATL IPv6 Test site)
>
For kicks I spun up an OVH VPS in BHS and here's the mtr to it:
Start: 2020-09-07T14:27:25-0400
HOST: work Loss% Snt Last Avg Best Wrst StDev
1. AS??? ap-ipv6-1494-udp.vpn (fd10:10:23::1) 0.0% 1 13.0 13.0 13.0 13.0 0.0
2. AS3842 2604:180::1 0.0% 1 22.9 22.9 22.9 22.9 0.0
3. AS2914 xe-0-0-25-1.a01.atlnga05.us.bb.gin.ntt.net (2001:418:0:5000::622) 0.0% 1 17.0 17.0 17.0 17.0 0.0
4. AS2914 ae-6.r23.atlnga05.us.bb.gin.ntt.net (2001:418:0:2000::22a) 0.0% 1 22.2 22.2 22.2 22.2 0.0
5. AS2914 ae-8.r04.atlnga05.us.bb.gin.ntt.net (2001:418:0:2000::10d) 0.0% 1 13.3 13.3 13.3 13.3 0.0
6. AS??? ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
7. AS1299 ash-bb2-v6.telia.net (2001:2000:3019:74::1) 0.0% 1 24.8 24.8 24.8 24.8 0.0
8. AS??? ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
9. AS16276 be100-1007.nwk-5-a9.nj.us (2607:5300::2) 0.0% 1 31.8 31.8 31.8 31.8 0.0
10. AS16276 be102.bhs-g2-nc5.qc.ca (2607:5300::1ce) 0.0% 1 46.5 46.5 46.5 46.5 0.0
11. AS16276 vl100.bhs-d2-a75.qc.ca (2607:5300::1cd) 0.0% 1 35.9 35.9 35.9 35.9 0.0
12. AS16276 be7.bhs-z2g2-a75.qc.ca (2607:5300::157) 0.0% 1 35.8 35.8 35.8 35.8 0.0
13. AS16276 po7.bhs-z2c2-a70.qc.ca (2607:5300:0:1:1:c2:2:0) 0.0% 1 36.5 36.5 36.5 36.5 0.0
14. AS16276 2607:5300:201:c2::14 0.0% 1 37.1 37.1 37.1 37.1 0.0
15. AS??? ??? 100.0 1 0.0 0.0 0.0 0.0 0.0
16. AS16276 vps-f0611836.vps.ovh.ca (2607:5300:201:3100::375d) 0.0% 1 35.7 35.7 35.7 35.7 0.0
-Jim P.
More information about the Outages-discussion
mailing list