[c-nsp] Router memory fragmentation error then reboot

[Paul Stewart]

Hi Jeff...

What IOS is this 7500 running?  We have had similar issues with some
older 2924 switches where massive amounts of DEAD processes would pop up
and crash them due to a memory leak.  I can't recall seeing this on a
7500 though.... 

Here's some output from a 7513 here for reference (no BGP though):

dis1-rtr-mb#show processes memory sorted allocated

Total: 226924128, Used: 20890824, Free: 206033304
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
 106   0 1474238412       3512      12828          0          0 PDU
DISPATCHER
  91   0 1273425604 1273405648      21948          0          0 Key Proc
 105   0  839124204  830051456      18580          0          0 IP SNMP
  49   0  507316004    4186640      25408     381780          0 IP Input
 111   0  392296720  307784976     200864          0          0 OSPF
Router
   0   0  207987496  307430904    4496992    5544788          0 *Dead*
  61   0  142743228    1198880      14432          0          0 TCP
Protocols
   6   0  114572728    4535220      82700  109789020  110473888 Pool
Manager
 107   0   79910816 1878745740      13108          0          0 SNMP
ENGINE
  50   0   67034332   67029196      11376          0          0 CDP
Protocol
  34   0   37888380   37888380       3828          0          0 Slave
Time
  36   0    4571096    4566852      11080          0          0 SSH
Event handle

If I recall correctly (someone may correct me on the list) that *Dead*
processes can be reclaimed for further usage so that may not be a
problem at all....  I'm guessing that either the router is actually
running out of memory due to the BGP tables or that there's a bug in the
IOS version....

All the best,

 
Paul Stewart
IP Routing/Switching
Nexicom Inc.
http://www.nexicom.net


-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Jeff Crowe
Sent: Friday, January 20, 2006 8:03 AM
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] Router memory fragmentation error then reboot

Hi all,

I have a Cisco 7513 in a production environment that is rebooting with
memory allocation or memory fragmentation errors.  The time between
reboots can be anywhere from days to months.  The router has 2 RSP4's
with 256MB ram each and 3 FEIP2 controllers (6 FastEthernet).  I have 3
upstream providers sending full BGP feeds connected to FE ports and run
access lists (51 rules and 73 rules respectively).  The router connects
to our core network via FE and does not have any VLANs enabled.

Here are the log entries just before a reboot:

Jan 20 00:57:28 router 78861: Pool: Processor  Free: 334728  Cause:
Memory fragmentation Jan 20 00:57:28 router 78862: Alternate Pool: None
Free: 0  Cause: No Alternate pool Jan 20 00:57:28 router 78863:
Jan 20 00:57:28 router 78864: -Process= "BGP Router", ipl= 0, pid= 154
Jan 20 00:57:28 router 78865: -Traceback= 40468F78 4046AF48 40464258
4097D7E4 4098AD1C 4098F54C 40990154 409808B8 409870C8 Jan 20 00:57:35
router 78867: 2w1d: %BGP-5-ADJCHANGE: neighbor a.a.a.a Down No memory
Jan 20 00:57:35 router 78868: 2w1d: %BGP-5-ADJCHANGE: neighbor b.b.b.b
Down No memory Jan 20 00:57:35 router 78869: 2w1d: %BGP-5-ADJCHANGE:
neighbor c.c.c.c Down No memory

then the router reboots.

When the router has been running for a bit, I see a large amount of
memory being consumed by the *DEAD* process.

#show processes memory sorted allocated
Total: 218963488, Used: 218059404, Free: 904084
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
 154   0  157700432     471692  154356368          0          0 BGP
Router
  72   0   39975556    2981460   39985400          0          0 IP RIB
Update
   0   0   29994656    8254384      83216    5544788          0 *Dead*
  62   0   16726320          0     704172          0          0 IP Input
  33   0    1310940    1048752     272032          0          0
Per-Second
Jobs

[snip]

uptime for the router is 6 hours, 56 minutes.

I have added "logging buffered 16000" & "logging history size 100" to
the router to see if this will help, but if anyone has suggestions how
to minimize memory usage, or find the source of the *DEAD* memory, that
would be appreciated.

Thanks,

Jeff.

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/