[c-nsp] Router memory fragmentation error then reboot

[Jeff Crowe]

Hi all,

I have a Cisco 7513 in a production environment that is rebooting with
memory allocation or memory fragmentation errors.  The time between reboots
can be anywhere from days to months.  The router has 2 RSP4's with 256MB ram
each and 3 FEIP2 controllers (6 FastEthernet).  I have 3 upstream providers
sending full BGP feeds connected to FE ports and run access lists (51 rules
and 73 rules respectively).  The router connects to our core network via FE
and does not have any VLANs enabled.

Here are the log entries just before a reboot:

Jan 20 00:57:28 router 78861: Pool: Processor  Free: 334728  Cause: Memory
fragmentation
Jan 20 00:57:28 router 78862: Alternate Pool: None  Free: 0  Cause: No
Alternate pool
Jan 20 00:57:28 router 78863:
Jan 20 00:57:28 router 78864: -Process= "BGP Router", ipl= 0, pid= 154
Jan 20 00:57:28 router 78865: -Traceback= 40468F78 4046AF48 40464258
4097D7E4 4098AD1C 4098F54C 40990154 409808B8 409870C8
Jan 20 00:57:35 router 78867: 2w1d: %BGP-5-ADJCHANGE: neighbor a.a.a.a Down
No memory
Jan 20 00:57:35 router 78868: 2w1d: %BGP-5-ADJCHANGE: neighbor b.b.b.b Down
No memory
Jan 20 00:57:35 router 78869: 2w1d: %BGP-5-ADJCHANGE: neighbor c.c.c.c Down
No memory

then the router reboots.

When the router has been running for a bit, I see a large amount of memory
being consumed by the *DEAD* process.

#show processes memory sorted allocated
Total: 218963488, Used: 218059404, Free: 904084
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
 154   0  157700432     471692  154356368          0          0 BGP Router
  72   0   39975556    2981460   39985400          0          0 IP RIB
Update
   0   0   29994656    8254384      83216    5544788          0 *Dead*
  62   0   16726320          0     704172          0          0 IP Input
  33   0    1310940    1048752     272032          0          0 Per-Second
Jobs

[snip]

uptime for the router is 6 hours, 56 minutes.

I have added "logging buffered 16000" & "logging history size 100" to the
router to see if this will help, but if anyone has suggestions how to
minimize memory usage, or find the source of the *DEAD* memory, that would
be appreciated.

Thanks,

Jeff.