[Outages-discussion] RoadRunner/Bright House cablemodem problems

Mon Sep 16 18:38:16 EDT 2013

The only thing I can think of, re: someone at the CMTS head seeing
"devices drop off the local Ethernet link" (I'm assuming that refers to
the LAN ports on the Motorola CM, and that he was SNMP querying the OIDs
associated with the switch portion of that CM), would be some kind of
PHY compatibility issue with auto-neg (speed/duplex/flow control).

More specific: this would be a PHY compatibility problem between the
PHY used for the LAN ports (switch) on the Motorola CM, and the PHY used
on the Fortigate.  The microcode/firmware on these devices that work
with the PHY are sometimes designed idiotically.

Residential example, but true story: when I upgraded from an Asus RT-N16
router H/W revision A0 to H/W revision A1, I suddenly noticed that only
100mbit/full was being negotiated between my SB6120 (H/W revision 1.0)
and my Asus.  Rolled back to the A0, gigE got negotiated.

Issue was alleviated when I replaced the SB6120 I had with a newer
hardware version (H/W revision 3.0), which used a completely different
vendor of NIC/PHY.  So even in today's world of gigE-over-copper, there
are still vendor PHY incompatibilities.  Sad.

Alternately -- and this would require you to look at logs or visual
indicators when it happens:

The CM might be rebooting (watchdog firing) due to something going on
pertaining to SNR or power levels on some channels, especially if DOCSIS
3.x is involved.  Frank has touched base on the deep internals of DOCSIS
protocol (which I do not know, I only read them when needed), but I do
remember there being a fiasco regarding channel bonding and, in layman's
terms, "what to do if a single channel flakes out" on some models of
Motorola devices.  For the SB6120 series there was an actual firmware
update pushed out to help address this.  I can dig up the release notes
if need be -- I participated in a thread on DSLR/BBR discussing this,
actually, which is why I remember it.

If it IS rebooting, then the issue is probably on the RF side.  DOCSIS
3.x is great on paper, but reality shows that certain kinds of
interference/noise can be introduced into the spectrum anywhere between
you and the CMTS (colloquially known as "the node") which will affect
you even more than if you lacked bonding.  Say you're bonding 8
downstream channels (common with Zoom modems) and 2 of them have SNR
issues, intermittently flaking out.  Depending on the firmware and how
it was written, this could cause the modem to act oddly.  I've seen this
in the wild as well.

If the device is rebooting, my first question would be: what frequency
ranges are being used for your downstream and upstream channels?  If
they're in the 700MHz+ range, then I smile -- guess what uses 700MHz+
as well?  Cellular LTE services (AT&T, Verizon, etc.).  This is the
issue I ran into that took Comcast 5 months to deal with (and a few
months after the cable runs got replaced, they actually changed their
CMTS to use 561MHz and up).

Maybe post some signal levels?  Your device will only have downstream
power + SNR and upstream only power (upstream SNR can be provided by
your cable provider), but that's good enough.  Oh, and if the modem
tracks signal stats (correctable vs. uncorrectable codewords), please
provide that too.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

On Mon, Sep 16, 2013 at 05:44:04PM -0400, Jay Ashworth wrote:
> I can get you the details on models and such tomorrow, but the topo
> is RF -> Cablemodem, Gig-E -> Fortigate 60D WAN1 port.  No other devices,
> in production; I had things plugged into the other ports only for 
> diagnosis.
> 
> That router has a /31 static configured on the wan port; the cablemodem
> will hand out NATted DHCP addresses, but they're 172.16/12.
> 
> ----- Original Message -----
> > From: "Jeremy Chadwick" <jdc at koitsu.org>
> > To: "Jay Ashworth" <jra at baylink.com>
> > Cc: "outages-discussion" <outages-discussion at outages.org>
> > Sent: Monday, September 16, 2013 5:16:09 PM
> > Subject: Re: [Outages-discussion] RoadRunner/Bright House cablemodem problems
> > On Mon, Sep 16, 2013 at 04:11:15PM -0400, Jay Ashworth wrote:
> > > On a new install for a client, we're seeing that the cable modem
> > > drops
> > > out on us, and in a very interesting way.
> > >
> > > Every so randomly often, it apparently does an uncommanded reboot
> > > (this
> > > according to a guy who was logged into the CMTS looking at it one
> > > time it
> > > did so), and when it does, 2 connected devices see the ethernet drop
> > > for 3-5
> > > seconds, come back up for 15, then drop again for 3-5, and then go
> > > back
> > > to normal. I see this behavior logged exactly to the second on both
> > > the
> > > attached Fortigate router and my Linux laptop; we've replaced the
> > > cable
> > > modem and the ethernet cables coming out of it.
> > >
> > > Oddly, it only seems to happen when there's traffic on the link; I
> > > haven't
> > > seen a drop get logged when the link is idle (overnight/weekends).
> > >
> > > The cablemodem is a DOCSIS3, Motorola Surfboard Extreme (that's not
> > > exact,
> > > but I can provide a model number if anyone thinks it's critical).
> > > Everything
> > > in the cage is on a brand new 2000VA UPS.
> > >
> > > Anyone ever seen a cablemodem misbehave in this particularly
> > > peculiar way
> > > before?
> > 
> > I can probably help (I just spent nearly 5 months getting Comcast to
> > deal with replacing some actual runs of cable between poles that were
> > accepting interference -- in the end things were successful), but I
> > need the following things:
> > 
> > - Exact model of Motorola modem (SBxxxx) you're using and its firmware
> > *and* hardware version (visible via http://192.168.100.1/cmHelp.htm)
> > - Whether DOCSIS 3.x is in use or not -- no, actually, i'll redact
> > that
> > and just request this: a screenshot or PDF of the /cmSignal.htm page
> > - Same as before, but for /cmLogs.htm
> > - A topology diagram of things (don't need IPs/etc., just general
> > what's-connected-to-what) -- I'm mainly interested in the device which
> > hooks up to your cable modem's Ethernet port. You mention "4 ports"
> > (discussed with Frank) which is a device I haven't used before; I'm
> > used to the consumer models which have 1 Ethernet port. Knowing
> > what's connected (brand/device/driver-wise) to all those Ethernet
> > ports would be of interest
> > 
> > If this turns out to be what my issue was (very hard to track down),
> > I can give you a perl script that I wrote that polls the cable modem
> > periodically and stores SNRs in CSV which you can graph using dygraphs
> > (very very easy, I can include a working .html and relevant .js file).
> > I can put up a demo for you online (with real data) if you want to see
> > it first.
> > 
> > I do have a gut feeling about what the issue is, but a topology
> > diagram
> > would help to rule out that feeling.
> > 
> > --
> > | Jeremy Chadwick jdc at koitsu.org |
> > | UNIX Systems Administrator http://jdc.koitsu.org/ |
> > | Making life hard for others since 1977. PGP 4BD6C0CB |
> 
> -- 
> Jay R. Ashworth                  Baylink                       jra at baylink.com
> Designer                     The Things I Think                       RFC 2100
> Ashworth & Associates     http://baylink.pitas.com         2000 Land Rover DII
> St Petersburg FL USA               #natog                      +1 727 647 1274