[Outages-discussion] [outages] Nationwide Comcast outages due to botched firmware upgrade

[Aaron C. de Bruyn]

File that under "very bad timing". ;)

-A

On Wed, Sep 30, 2020 at 10:09 AM Andrew Gallo <akg1330 at gmail.com> wrote:

> I wonder if this is what caused the disruption to the FCC's Live Stream
> of it's Sept 2020 meeting
>
> From the transcript:
> > JOSH: WE ARE ATTEMPTING TO MOVE TO A SECONDARY STREAM. I'M NOT SURE THAT
> WE ARE GOING TO BE ABLE TO GET THE LIVESTREAM BACK UP AGAIN. WE ARE ABLE TO
> RECORD THE MEETING. WE WILL ATTEMPT ONE MORE TIME. APPARENTLY COMCAST IS
> UPGRADING MY SERVICE AND I'M DESPERATELY TRYING TO RECYCLE THE MODEM SO IT
> RECOGNIZES IT. RIGHT NOW I AM NOT GETTING CONNECTIVITY THROUGH TO YOUTUBE.
> STEVE IS RUNNING INTO ISSUES GETTING THROUGH. IF YOU WILL BEAR WITH ME, IT
> IS UP TO USA HOW TO CONDUCT THE MEETING. I'M AT A LOSS FOR ANOTHER WAY OF
> GETTING THIS TO YOUTUBE AT THE MOMENT.
>
>
>
>
>
> On 9/30/2020 11:25 AM, Rusty Dekema wrote:
> > That's interesting! I have Comcast "business" service at home with a
> > DPC3941B gateway, and I did have an unexplained service interruption
> > last night just before midnight.
> >
> > I already power cycled the device, and it is now running firmware
> > version dpc3941b-v303r20421771-190327a-CMCST.
> >
> > Do you happen to know the previous firmware version and/or the version
> > they were attempting to push out?
> >
> > These devices could certainly benefit from some firmware improvements.
> > The last time I checked (which may have been on a previous release),
> > the static IPv4 routes that the web UI lets you enter do not actually
> > take effect (unlike in previous Comcast gateway models). This is
> > annoying, but (in my setup) can be worked around by using proxy arp on
> > the downstream router. There is also some flakiness with DHCPv6-PD
> > which causes the device to stop routing traffic to the delegated
> > prefix(es) after several days and requires a reset of the gateway to
> > correct. (A renew/rebind from the DHCPv6 client is not sufficient to
> > clear the problem.)
> >
> > Did you hear anything about plans to re-push the new version [or a
> > patched version thereof]?
> >
> > Cheers,
> > Rusty
> >
> >
> > On Wed, Sep 30, 2020 at 10:56 AM Cary Wiedemann via Outages
> > <outages at outages.org> wrote:
> >>
> >> If you noticed your Comcast sites all over the country were down last
> night it was due to a botched firmware upgrade pushed to many Cisco
> modems.  Comcast Business uses Cisco DPC3941B modems in many regions and
> these were particularly affected.
> >>
> >> Comcast can't reboot these modems remotely anymore.  They all need to
> be physically power cycled and will downgrade their firmware to an
> operational version during the boot process.
> >>
> >> I've had modems impacted from coast to coast, but not in every region.
> The first wave of outages began yesterday 9/30 at 5:47pm EDT, but many
> other sites didn't experience their first outage until well after midnight.
> >>
> >> Just an FYI if anyone else saw their branch networks in shambles this
> morning.
> >>
> >> The Comcast master "service incident" number is: SI045791942
> >>
> >> - Cary
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> > _______________________________________________
> > Outages-discussion mailing list
> > Outages-discussion at outages.org
> > https://puck.nether.net/mailman/listinfo/outages-discussion
> >
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20200930/0216c7bb/attachment.htm>