[outages] Spectrum/Charter IPv6 connectivity to CloudFlare down in Texas

[Daniel Marks]

As a follow up, we're still working with Spectrum engineers on a
workaround, IPv6 is still mostly down in the DFW area on Spectrum
outside of the large CDNs (i.e. Quad9 and Telia-bound traffic is still
offline) - their ETA is another 24-48 hours as a line card for Houston
was still on order, according to an engineer I spoke to. Not a great
time for us IPv6-only networks.

On Thu, Apr 20, 2023 at 11:18 AM Tom Strickx <tstrickx at cloudflare.com> wrote:
>
> Charter has acknowledged the issue, and is tracking this internally as INC000026440632. Appears to be an issue with a linecard in Houston.
>
> We've rerouted IPv6 out of Dallas for now to mitigate impact. Sorry for the trouble!
> --
> Tom Strickx
> Principal Network Engineer
> AS13335 - Cloudflare
>
>
> On Thu, Apr 20, 2023 at 2:05 PM Tom Strickx <tstrickx at cloudflare.com> wrote:
>>
>> Hey folks,
>>
>> Could either of you contact me off-list so I can investigate? A specific start-of-problem would be super helpful.
>> --
>> Tom Strickx
>> Principal Network Engineer
>> AS13335 - Cloudflare
>>
>>
>> On Thu, Apr 20, 2023 at 4:38 AM Daniel Marks via Outages <outages at outages.org> wrote:
>>>
>>> Have been dealing with this for 2 days, nobody at Spectrum can figure this out.
>>>
>>> Daniel Marks
>>> May Mobility, Inc
>>>
>>> >
>>> > On Apr 19, 2023, at 19:26, Carl Perry via Outages <outages at outages.org> wrote:
>>> >
>>> > Things have been rather slow the last couple of days, just figured out why; No IPv6 connectivity to any CloudFlare services.
>>> >
>>> > > mtr -6 --report-wide cloudflare.com
>>> > Start: 2023-04-19T23:20:42+0000
>>> > HOST: defiant Loss%   Snt   Last   Avg  Best  Wrst StDev
>>> >   1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com 0.0%    10    3.6  19.1   1.5  93.2  36.1
>>> >   2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0%    10   14.5  36.6  11.3 171.2  50.0
>>> >   3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0%    10   18.8  37.3  12.0 143.9  45.3
>>> >   4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0%    10   65.8  30.2  11.7  65.8  24.9
>>> >   5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0%    10   26.5  25.5  24.5  26.5   1.4
>>> >   6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0%    10   25.7  25.7  25.7  25.7   0.0
>>> >   7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0%    10   25.4  26.3  18.2  44.4   9.5
>>> >   8.|-- ??? 100.0    10    0.0   0.0   0.0   0.0   0.0
>>> >
>>> > IPv4 works however:
>>> >
>>> > > mtr -4 --report-wide cloudflare.com
>>> > Start: 2023-04-19T23:20:21+0000
>>> > HOST: defiant                                     Loss%   Snt Last   Avg  Best  Wrst StDev
>>> >   1.|-- _gateway                                     0.0%    10 2.1   1.8   1.4   2.1   0.2
>>> >   2.|-- cpe-70-114-160-1.austin.res.rr.com           0.0%    10 51.9  34.4  14.3 149.7  42.1
>>> >   3.|-- lag-63.hcr01rdrktxaz.netops.charter.com      0.0%    10 13.1  38.5  13.1 159.1  44.6
>>> >   4.|-- lag-19.ausutxla01r.netops.charter.com        0.0%    10 18.1  21.0  14.3  34.8   6.0
>>> >   5.|-- lag-22.rcr01dllatx37.netops.charter.com      0.0%    10 23.4  23.3  18.3  33.3   4.3
>>> >   6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com  0.0%    10 19.8  27.0  16.1  51.2  11.9
>>> >   7.|-- lag-0.pr3.dfw10.netops.charter.com           0.0%    10 39.3  28.5  14.1  62.3  16.7
>>> >   8.|-- 141.101.74.104                              40.0%    10 53.2  44.3  19.4 120.2  39.2
>>> >   9.|-- 172.71.168.2                                 0.0%    10 28.3  28.8  10.8  79.4  19.5
>>> >  10.|-- 104.16.133.229                               0.0%    10 122.0  42.8  16.9 122.0  34.4
>>> >
>>> > It's kind of wild how many things don't fall back to IPv4 quickly (lots of Android TV apps) or at all (.NET). I tried reaching out to Spectrum support, but being a lowly consumer I was only told that "The modem doesn't have any filters" and that "the problem may be on the CloudFlare side", which was sadly what I predicted. So, I figured I would raise the issue here in the hopes that actual network people would see it.
>>> >
>>> >   -Carl
>>> >
>>> >
>>> > _______________________________________________
>>> > Outages mailing list
>>> > Outages at outages.org
>>> > https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outages&source=gmail-imap&ust=1682551587000000&usg=AOvVaw0omaxNXXnf3Ff4dHAQDhA8
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages



-- 
-Dan