[Outages-discussion] [outages] Nationwide Comcast outages due to botched firmware upgrade

[Rusty Dekema]

That's interesting! I have Comcast "business" service at home with a
DPC3941B gateway, and I did have an unexplained service interruption
last night just before midnight.

I already power cycled the device, and it is now running firmware
version dpc3941b-v303r20421771-190327a-CMCST.

Do you happen to know the previous firmware version and/or the version
they were attempting to push out?

These devices could certainly benefit from some firmware improvements.
The last time I checked (which may have been on a previous release),
the static IPv4 routes that the web UI lets you enter do not actually
take effect (unlike in previous Comcast gateway models). This is
annoying, but (in my setup) can be worked around by using proxy arp on
the downstream router. There is also some flakiness with DHCPv6-PD
which causes the device to stop routing traffic to the delegated
prefix(es) after several days and requires a reset of the gateway to
correct. (A renew/rebind from the DHCPv6 client is not sufficient to
clear the problem.)

Did you hear anything about plans to re-push the new version [or a
patched version thereof]?

Cheers,
Rusty


On Wed, Sep 30, 2020 at 10:56 AM Cary Wiedemann via Outages
<outages at outages.org> wrote:
>
> If you noticed your Comcast sites all over the country were down last night it was due to a botched firmware upgrade pushed to many Cisco modems.  Comcast Business uses Cisco DPC3941B modems in many regions and these were particularly affected.
>
> Comcast can't reboot these modems remotely anymore.  They all need to be physically power cycled and will downgrade their firmware to an operational version during the boot process.
>
> I've had modems impacted from coast to coast, but not in every region.  The first wave of outages began yesterday 9/30 at 5:47pm EDT, but many other sites didn't experience their first outage until well after midnight.
>
> Just an FYI if anyone else saw their branch networks in shambles this morning.
>
> The Comcast master "service incident" number is: SI045791942
>
> - Cary
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages