[c-nsp] 3750 stack "inaccessible" - memory leak

[Nathanael Law]

Hello Jeff,

This may all be unrelated, since a large variety of memory leaks may cause
this behavior, but here is a summary of a recent case we had with similar
symptoms.

We've had a similar issue with a stack of WS-C3750E-48TDs running 15.0(2)SE1.
There is a memory leak in the device sensor which affects at lease the Auth
Manager and CDP Protocol processes.  Running "no macro auto monitor" stops
the memory growth of these processes, but will not free up memory that has
already leaked.  The leak may also affect LLDP, SIP, and DHCP snooping, but
we aren't running any of those on our affected switch stack, so I can't
comment on processes directly related to those features.

Symptomatically, once memory was exhausted, we could only log in via console.
Attempts to log in via SSH resulted in various errors including
%AAA-3-ACCT_LOW_MEM_UID_FAIL.  Note that this error may not appear even if
you are affected by the bug.  "show proc mem sort" should be a fairly good
indicator.  The holding values are the ones to take note of (I've marked
them with ## ## in the output below).

xx-xxxxx-xxx-x-xxx#rem com all show proc mem sort
Switch : 1 : (Master)
---------------------

Processor Pool Total:  175321924 Used:  167405032 Free:    7916892
      I/O Pool Total:   16777216 Used:   12876652 Free:    3900564
Driver te Pool Total:    4194304 Used:     106740 Free:    4087564

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
 210   0 1543415888  792708252 ##94566048##    25380          0 Auth Manager
   0   0  103488488   42309088   56639052          0          0 *Init*
 213   0  480837472  146755172 ##12896400##    10152          0 CDP Protocol
   0   0  656042836  785140364    6873320   14305647    1472786 *Dead*
  93   0    1878040     698288    1004544       5076          0 Stack Mgr Notifi

TAC has identified the bug in CSCub85948.  Per TAC, there is no ETA on the
release of a fixed IOS version, but it is currently slated for 15.0(2)SE3.

They also recommend running the following to work-around the leak, but we
found these commands unnecessary:
(config)#device-sensor filter-spec cdp exclude all
(config)#device-sensor filter-spec dhcp exclude all
(config)#device-sensor filter-spec lldp exclude all
(config)#device-sensor filter-spec sip exclude all

If you'd like me to send your our SR#, or other details of our case,
please let me know.

Best regards,

Nathanael Law


> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-
> bounces at puck.nether.net] On Behalf Of Jeff Kell
> Sent: Thursday, April 04, 2013 20:46
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] 3750 stack "inaccessible" - memory leak
> 
> Each time my patience wears thin with TAC, I come here; it's typically
> more productive :)
> 
> February 2011:  3750G stack stops answering to telnet/ssh with timeouts,
> serial console reports "%% Low on memory; try again later".
> 
> Gets blamed on CSCsu27706, fixed in 12.2(44); but we're now on 15.0(2).
> And having the same exact symptoms again.
> 
> Anyone been there, done that, got the T-shirt?  Better yet, got a
> workaround or fix?
> 
> Jeff
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/