[c-nsp] ASR920 randomly loosing layer-2 on a port

[Gert Doering]

Hi,

On Mon, Jul 11, 2022 at 04:47:57PM +0000, Brian Turnbow wrote:
> > On Mon, Jul 11, 2022 at 03:59:02PM +0000, Brian Turnbow wrote:
> > > Yep, sounds like the infamous uptime over 2 years "feature" from 3.16
> > (something)..
> > > Reboot and upgrade was the only way we fixed it....
> > 
> > Uh.  Could you elaborate on what that "feature" is, exactly?
> 
> It was the bug where after after two years of uptime
> If an interface went down it would stick as up and not pass traffic
> You could not provision new interfaces.
> Counters also stopped working. (we used this to find affected units)

Now THAT is interesting.  I'm a bit further distanced from day-to-day
operations these days (otherwise I might have noticed), but indeed,
counters didn't work anymore either.  "No traffic on this box!" which
I know to be not true (our daily TSM backups go through there...) - and
after reboot, "Traffic!".

Very interesting.  "Interface itself" counters are all "0", but service
instance counters (gi0/0/2 si 90) still show traffic.  So that's actually
something our alarming could trigger on "si has > 1 Mbit, interface itself
has 0"...

[..]
> Sounds like it may be different.
> Did the counters work?
> Maybe they decided to add it into 16.06 , you never know what a BU may decide is a must have feature....

Obviously, 16.06 has much improved performance, so 2-year-bugs are now 
hit after 0.5 years already!

OTOH... seems it wasn't actually 27 weeks uptime, but quite a bit more,
which was just distorted by SNMP uptime wrapping (and our prometheus
instance not properly distinguishing this for old data, it only recently
learned to query that other OID).

So, definitely more than 2 years, and traffic counters stopped some 5 months
ago...  and we did not try to actually bring up anything new since then.

Yeah, thanks a lot for this information.  This will be very helpful to
avoid needless frustration by our on-site people ("it does not link! can
you please try a different cable?  did you get the patch right?").

gert

-- 
"If was one thing all people took for granted, was conviction that if you 
 feed honest figures into a computer, honest figures come out. Never doubted 
 it myself till I met a computer with a sense of humor."
                             Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany                             gert at greenie.muc.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 630 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-nsp/attachments/20220711/9985735d/attachment.sig>