[Outages-discussion] [outages] Nationwide Comcast outages due to botched firmware upgrade

[Andrew Gallo]

I wonder if this is what caused the disruption to the FCC's Live Stream
of it's Sept 2020 meeting

From the transcript:
> JOSH: WE ARE ATTEMPTING TO MOVE TO A SECONDARY STREAM. I'M NOT SURE THAT WE ARE GOING TO BE ABLE TO GET THE LIVESTREAM BACK UP AGAIN. WE ARE ABLE TO RECORD THE MEETING. WE WILL ATTEMPT ONE MORE TIME. APPARENTLY COMCAST IS UPGRADING MY SERVICE AND I'M DESPERATELY TRYING TO RECYCLE THE MODEM SO IT RECOGNIZES IT. RIGHT NOW I AM NOT GETTING CONNECTIVITY THROUGH TO YOUTUBE. STEVE IS RUNNING INTO ISSUES GETTING THROUGH. IF YOU WILL BEAR WITH ME, IT IS UP TO USA HOW TO CONDUCT THE MEETING. I'M AT A LOSS FOR ANOTHER WAY OF GETTING THIS TO YOUTUBE AT THE MOMENT.





On 9/30/2020 11:25 AM, Rusty Dekema wrote:
> That's interesting! I have Comcast "business" service at home with a
> DPC3941B gateway, and I did have an unexplained service interruption
> last night just before midnight.
> 
> I already power cycled the device, and it is now running firmware
> version dpc3941b-v303r20421771-190327a-CMCST.
> 
> Do you happen to know the previous firmware version and/or the version
> they were attempting to push out?
> 
> These devices could certainly benefit from some firmware improvements.
> The last time I checked (which may have been on a previous release),
> the static IPv4 routes that the web UI lets you enter do not actually
> take effect (unlike in previous Comcast gateway models). This is
> annoying, but (in my setup) can be worked around by using proxy arp on
> the downstream router. There is also some flakiness with DHCPv6-PD
> which causes the device to stop routing traffic to the delegated
> prefix(es) after several days and requires a reset of the gateway to
> correct. (A renew/rebind from the DHCPv6 client is not sufficient to
> clear the problem.)
> 
> Did you hear anything about plans to re-push the new version [or a
> patched version thereof]?
> 
> Cheers,
> Rusty
> 
> 
> On Wed, Sep 30, 2020 at 10:56 AM Cary Wiedemann via Outages
> <outages at outages.org> wrote:
>>
>> If you noticed your Comcast sites all over the country were down last night it was due to a botched firmware upgrade pushed to many Cisco modems.  Comcast Business uses Cisco DPC3941B modems in many regions and these were particularly affected.
>>
>> Comcast can't reboot these modems remotely anymore.  They all need to be physically power cycled and will downgrade their firmware to an operational version during the boot process.
>>
>> I've had modems impacted from coast to coast, but not in every region.  The first wave of outages began yesterday 9/30 at 5:47pm EDT, but many other sites didn't experience their first outage until well after midnight.
>>
>> Just an FYI if anyone else saw their branch networks in shambles this morning.
>>
>> The Comcast master "service incident" number is: SI045791942
>>
>> - Cary
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20200930/142543f1/attachment.sig>