[c-nsp] 6504-E IOS SSH/memory issues

[Patrick M. Hausen]

Hi, all,

in Saturday our Rancid started to complain that it could not log on to one
of our core/uplink routers, anymore. Yet the system is generally alive and
happily pushing packets - Nagios did not ring me about any link or service
failing, so this came as a bit of a surprise.

Turns out, SSH logins are not possible, anymore. Telnet and rsh work just
fine. For each faile SSH login there is a line like this in the log:

Mar 20 12:30:09.415: %AAA-3-ACCT_LOW_MEM_UID_FAIL: AAA unable to create UID for incoming calls due to insufficient processor memory

Ah ... OK ... if it's failing in AAA, why does telnet still work? And the free memory
doesn't look too bad, either:

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   477267E0   881661984   860385044    21276940    18235288    20933772
      I/O    8000000    67108864    21605604    45503260    45451176    45501532

          Processor memory

Alloc PC        Size     Blocks      Bytes    What

0x4014A218 0000000024 0000000001 0000000024    XDR: mfib pltf group
0x4014A218 0000000028 0000000001 0000000028    XDR: mfib pltf group
0x4014A218 0000000032 0000000001 0000000032    XDR: mfib pltf group
0x401567F4 0000003808 0000000001 0000003808    Init
0x4016D4BC 0000000024 0000000001 0000000024    Init
...

In the thousands of lines that follow, there are precisely 256 memory blocks
allocated to the "SSH process". Is this a single process holding all that memory
or are there 256 SSH processes, that are somewhat stuck/zombie because
they are not terminated when the connection is closed?

I admit that I rarely log off, but rather just close the window running my SSH connection.
Bad admin. ;-) But any sane OS should timeout the TCP connection eventually and
then terminate the process waiting on that socket.

IOS version is 15.1(2)SY1 advanced enterprise.

How can I proceed finding and eliminating the root cause? Rebooting the box to clean
up is an option if planned ahead, but not a suitable permanent fix (i.e. rebooting regularly
is out of the question).

Thanks for any hints,
Patrick
-- 
punkt.de GmbH * Kaiserallee 13a * 76133 Karlsruhe
Tel. 0721 9109 0 * Fax 0721 9109 100
info at punkt.de       http://www.punkt.de
Gf: Jürgen Egeling      AG Mannheim 108285