<div dir="ltr"><div dir="ltr">On my end, it seems that it's not only Cloudflare, I also can't reach Quad9 or our AWS EC2 instances at us-east-1 over IPv6. Google and Facebook seem to be reachable over IPv6, though.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 20, 2023 at 1:23 AM Jeremy Chadwick via Outages-discussion <<a href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">(Moving to -discussion because there's nothing concrete in what I'm<br>
about to say, i.e. nothing worth waking people up for.)<br>
<br>
1. Your mtrs have lost their monospace/whitespace formatting for<br>
whatever reason, making them difficult to read.<br>
<br>
2. Your IPv6 mtr looks like it indicates POSSIBLE packet loss starting<br>
at hop #4 and seems to trickle down through hops #5/6/7. I can't be<br>
100% certain the loss is real (it could be ICMP deprio on Charter's<br>
routers, but only for IPv6?!) because I have not seen what a working v6<br>
mtr from you which uses the same routers/path. If it IS real loss, then<br>
the problem is much more likely within Spectrum/Charter's network, but I<br>
can't say that with absolute certainty, leading me to:<br>
<br>
2. You have no return-path mtr for either v4 or v6, so you only are able<br>
to see 50% of the entire picture. Asymmetric routing is commonplace<br>
these days, which is why you need it from both directions.<br>
<br>
If you really and absolutely think this is a CloudFlare problem (I am<br>
not sold on that given the above two points), then we continue with:<br>
<br>
3. Based on 141.101.74.104 in your v4 mtr, it looks like<br>
Spectrum/Charter has a peering agreement with CloudFlare.<br>
<br>
4. CloudFlare does not offer a public looking glass, but they do offer<br>
an HTTP API-based traceroute, though it is sub-par in quality and kind<br>
of bizarre, not to mention I don't know if it supports v6:<br>
<a href="https://blog.clucas.fr/2021/11/18/traceroute-from-cloudflare-without-looking-glass/" rel="noreferrer" target="_blank">https://blog.clucas.fr/2021/11/18/traceroute-from-cloudflare-without-looking-glass/</a><br>
<br>
Given items #3/4, have you tried reaching out to CloudFlare for a<br>
return-path mtr back to whatever your v6 source IP is?<br>
<br>
Their contact information is public: <a href="https://www.peeringdb.com/net/4224" rel="noreferrer" target="_blank">https://www.peeringdb.com/net/4224</a><br>
<br>
-- <br>
| Jeremy Chadwick <a href="http://jdc_at_koitsu.org" rel="noreferrer" target="_blank">jdc_at_koitsu.org</a> |<br>
| UNIX Systems Administrator PGP 0x2A389531 |<br>
| Making life hard for others since 1977. |<br>
<br>
On Wed, Apr 19, 2023 at 11:37:10PM -0400, Daniel Marks via Outages wrote:<br>
> Have been dealing with this for 2 days, nobody at Spectrum can figure this out. <br>
> <br>
> Daniel Marks<br>
> May Mobility, Inc<br>
> <br>
> > <br>
> > On Apr 19, 2023, at 19:26, Carl Perry via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br>
> > <br>
> > Things have been rather slow the last couple of days, just figured out why; No IPv6 connectivity to any CloudFlare services.<br>
> > <br>
> > > mtr -6 --report-wide <a href="http://cloudflare.com" rel="noreferrer" target="_blank">cloudflare.com</a><br>
> > Start: 2023-04-19T23:20:42+0000<br>
> > HOST: defiant Loss% Snt Last Avg Best Wrst StDev<br>
> > 1.|-- <a href="http://2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com" rel="noreferrer" target="_blank">2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com</a> 0.0% 10 3.6 19.1 1.5 93.2 36.1<br>
> > 2.|-- <a href="http://2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com" rel="noreferrer" target="_blank">2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com</a> 0.0% 10 14.5 36.6 11.3 171.2 50.0<br>
> > 3.|-- <a href="http://lag-63.hcr02rdrktxaz.netops.charter.com" rel="noreferrer" target="_blank">lag-63.hcr02rdrktxaz.netops.charter.com</a> 0.0% 10 18.8 37.3 12.0 143.9 45.3<br>
> > 4.|-- <a href="http://lag-19.ausxtxir02r.netops.charter.com" rel="noreferrer" target="_blank">lag-19.ausxtxir02r.netops.charter.com</a> 60.0% 10 65.8 30.2 11.7 65.8 24.9<br>
> > 5.|-- <a href="http://lag-22.rcr01hstqtx02.netops.charter.com" rel="noreferrer" target="_blank">lag-22.rcr01hstqtx02.netops.charter.com</a> 80.0% 10 26.5 25.5 24.5 26.5 1.4<br>
> > 6.|-- <a href="http://lag-16.hstqtx0209w-bcr00.netops.charter.com" rel="noreferrer" target="_blank">lag-16.hstqtx0209w-bcr00.netops.charter.com</a> 90.0% 10 25.7 25.7 25.7 25.7 0.0<br>
> > 7.|-- <a href="http://lag-12.dllstx976iw-bcr00.netops.charter.com" rel="noreferrer" target="_blank">lag-12.dllstx976iw-bcr00.netops.charter.com</a> 40.0% 10 25.4 26.3 18.2 44.4 9.5<br>
> > 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0<br>
> > <br>
> > IPv4 works however:<br>
> > <br>
> > > mtr -4 --report-wide <a href="http://cloudflare.com" rel="noreferrer" target="_blank">cloudflare.com</a><br>
> > Start: 2023-04-19T23:20:21+0000<br>
> > HOST: defiant Loss% Snt Last Avg Best Wrst StDev<br>
> > 1.|-- _gateway 0.0% 10 2.1 1.8 1.4 2.1 0.2<br>
> > 2.|-- <a href="http://cpe-70-114-160-1.austin.res.rr.com" rel="noreferrer" target="_blank">cpe-70-114-160-1.austin.res.rr.com</a> 0.0% 10 51.9 34.4 14.3 149.7 42.1<br>
> > 3.|-- <a href="http://lag-63.hcr01rdrktxaz.netops.charter.com" rel="noreferrer" target="_blank">lag-63.hcr01rdrktxaz.netops.charter.com</a> 0.0% 10 13.1 38.5 13.1 159.1 44.6<br>
> > 4.|-- <a href="http://lag-19.ausutxla01r.netops.charter.com" rel="noreferrer" target="_blank">lag-19.ausutxla01r.netops.charter.com</a> 0.0% 10 18.1 21.0 14.3 34.8 6.0<br>
> > 5.|-- <a href="http://lag-22.rcr01dllatx37.netops.charter.com" rel="noreferrer" target="_blank">lag-22.rcr01dllatx37.netops.charter.com</a> 0.0% 10 23.4 23.3 18.3 33.3 4.3<br>
> > 6.|-- <a href="http://lag-14.dllstx976iw-bcr00.netops.charter.com" rel="noreferrer" target="_blank">lag-14.dllstx976iw-bcr00.netops.charter.com</a> 0.0% 10 19.8 27.0 16.1 51.2 11.9<br>
> > 7.|-- <a href="http://lag-0.pr3.dfw10.netops.charter.com" rel="noreferrer" target="_blank">lag-0.pr3.dfw10.netops.charter.com</a> 0.0% 10 39.3 28.5 14.1 62.3 16.7<br>
> > 8.|-- 141.101.74.104 40.0% 10 53.2 44.3 19.4 120.2 39.2<br>
> > 9.|-- 172.71.168.2 0.0% 10 28.3 28.8 10.8 79.4 19.5<br>
> > 10.|-- 104.16.133.229 0.0% 10 122.0 42.8 16.9 122.0 34.4<br>
> > <br>
> > It's kind of wild how many things don't fall back to IPv4 quickly (lots of Android TV apps) or at all (.NET). I tried reaching out to Spectrum support, but being a lowly consumer I was only told that "The modem doesn't have any filters" and that "the problem may be on the CloudFlare side", which was sadly what I predicted. So, I figured I would raise the issue here in the hopes that actual network people would see it.<br>
> > <br>
> > -Carl<br>
> > <br>
> > <br>
> > _______________________________________________<br>
> > Outages mailing list<br>
> > <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
> > <a href="https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outages&source=gmail-imap&ust=1682551587000000&usg=AOvVaw0omaxNXXnf3Ff4dHAQDhA8" rel="noreferrer" target="_blank">https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outages&source=gmail-imap&ust=1682551587000000&usg=AOvVaw0omaxNXXnf3Ff4dHAQDhA8</a><br>
> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org" target="_blank">Outages-discussion@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages-discussion" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><font face="arial, sans-serif">-Dan</font></div></div></div>