[c-nsp] 3750: SNMP-3-INPUT_QFULL_ERR, ssh session dies, show tech support fails, switch stack crashes on reload

[Sebastian Beutel]

Hi List,

    we are running a lot of (three-digit) 3750 stacks that are showing a
strange phenomenon by and by. Meanwhile we have five affected switches that
had an uptime of almost two years and an old IOS 12.2(44)SE in common.
    The first indication so far is, that we see "%SNMP-3-INPUT_QFULL_ERR" in
the syslog for no reason (the affected switch receives as much snmp request
as any other switch on our network). If we ssh to the affected switch and do
a "show interfaces status" it shows a couple of interfaces of the first
switch, then the ssh session crashes. The same with "show etherchannel
summary". If we reconnect again a "show users" lists the broken connection
but a "clear line vty" does not reset it. If we do a "show tech-support |
redirect tftp:..." the ssh session also crashes. The file on the tftserver
ends with the interface at which "sho int statu" breaks.
    At this stage the stack still seems to forward traffic but if we do a
reload in a maintenance window things get worse: The switch that has been
the stack master instantly crashes and does not recover. Forwarding stops
and the management interface does not come back. Even the console is
unusable. The only thing to remedy the situation is to unplug the mains
cables. After that the switch comes back as though nothing had happened.

We understand that we need to update the IOS (and we did on some of them).
The problem though is that the reload that is associated with the update may
also cause outages. We suspect that stacks are unstable long before they
attract attention due do the snmp error messages. It would be a great help
to predict whether a switch will survive an update or otherwise would need
it's mains supply interrupted.
 

Thanks in advance,
                            Sebastian.

-- 
Dipl.-Ing. Sebastian Beutel                              tel: +49-711-685-64538
Rechenzentrum Universitaet Stuttgart        http://www.rus.uni-stuttgart.de/nks
Netze und Kommunikationssysteme              Allmandring 30A, D-70550 Stuttgart