[c-nsp] Router memory fragmentation error then reboot

[Oliver Boehmer (oboehmer)]

Jeff,

quick answer: get more memory or reduce the number of BGP paths. You are
running the router with less than one 1MB of free memory, which is
always bad for such type of applications (where memory consumption comes
and goes, routes are withdrawn and added). 256MB is just not enough
anymore with full feeds these days..

"Memory fragmentation" is a misleading log message in this situation,
the router tried to allicate 300k out of the 1MB (or how much was free
when the router did it), but it couldn't allocate a consecutive block of
memory of this size in the 1MB of free memory. No surprise..

And the *Dead* process is "holding" only 80k of memory.. check out
http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_not
e09186a00800a6f3a.shtml for explanations of *Dead*

	oli


Jeff Crowe <> wrote on Friday, January 20, 2006 2:03 PM:

> Hi all,
> 
> I have a Cisco 7513 in a production environment that is rebooting with
> memory allocation or memory fragmentation errors.  The time between
> reboots can be anywhere from days to months.  The router has 2 RSP4's
> with 256MB ram each and 3 FEIP2 controllers (6 FastEthernet).  I have
> 3 upstream providers sending full BGP feeds connected to FE ports and
> run access lists (51 rules and 73 rules respectively).  The router
> connects to our core network via FE and does not have any VLANs
> enabled. 
> 
> Here are the log entries just before a reboot:
> 
> Jan 20 00:57:28 router 78861: Pool: Processor  Free: 334728  Cause:
> Memory fragmentation
> Jan 20 00:57:28 router 78862: Alternate Pool: None  Free: 0  Cause: No
> Alternate pool
> Jan 20 00:57:28 router 78863:
> Jan 20 00:57:28 router 78864: -Process= "BGP Router", ipl= 0, pid= 154
> Jan 20 00:57:28 router 78865: -Traceback= 40468F78 4046AF48 40464258
> 4097D7E4 4098AD1C 4098F54C 40990154 409808B8 409870C8
> Jan 20 00:57:35 router 78867: 2w1d: %BGP-5-ADJCHANGE: neighbor
> a.a.a.a Down No memory
> Jan 20 00:57:35 router 78868: 2w1d: %BGP-5-ADJCHANGE: neighbor
> b.b.b.b Down No memory
> Jan 20 00:57:35 router 78869: 2w1d: %BGP-5-ADJCHANGE: neighbor
> c.c.c.c Down No memory
> 
> then the router reboots.
> 
> When the router has been running for a bit, I see a large amount of
> memory being consumed by the *DEAD* process.
> 
> #show processes memory sorted allocated
> Total: 218963488, Used: 218059404, Free: 904084
>  PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs
>  Process 154   0  157700432     471692  154356368          0         
>   0 BGP Router 72   0   39975556    2981460   39985400          0    
> 0 IP RIB 
> Update
>    0   0   29994656    8254384      83216    5544788          0 *Dead*
>   62   0   16726320          0     704172          0          0 IP
>   Input 33   0    1310940    1048752     272032          0          0
> Per-Second Jobs
> 
> [snip]
> 
> uptime for the router is 6 hours, 56 minutes.
> 
> I have added "logging buffered 16000" & "logging history size 100" to
> the router to see if this will help, but if anyone has suggestions
> how to minimize memory usage, or find the source of the *DEAD*
> memory, that would be appreciated.
> 
> Thanks,
> 
> Jeff.
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/