[outages] Nationwide Comcast outages due to botched firmware upgrade

[Cary Wiedemann]

If you noticed your Comcast sites all over the country were down last night
it was due to a botched firmware upgrade pushed to many Cisco modems.
Comcast Business uses Cisco DPC3941B modems in many regions and these were
particularly affected.

Comcast can't reboot these modems remotely anymore.  They all need to be
physically power cycled and will downgrade their firmware to an operational
version during the boot process.

I've had modems impacted from coast to coast, but not in every region.  The
first wave of outages began yesterday 9/30 at 5:47pm EDT, but many other
sites didn't experience their first outage until well after midnight.

Just an FYI if anyone else saw their branch networks in shambles this
morning.

The Comcast master "service incident" number is: SI045791942

- Cary
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20200930/6f7b8abb/attachment.htm>