[c-nsp] Wierd memory issue with SXI/SXI1 on 6500 w/ SUP720-3BXL [SOLVED]

[Chris Phillips]

A "show memory allocating-process totals" is very telling.

llocator PC Summary for: Processor
Displayed first 2048 Allocator PCs only

     PC          Total   Count  Name
0x4035A6C8  295186172    4420  BGP battr chun
0x40809510   45688268     695  CEF: fib

BGP is definitely the culprit.  It has consumed almost 250 million more 
bytes than the next closest process.  Yikes!

Looking at a "healthy" router, this process is just under the 100 
million byte mark.

I received an off-list reply that contained the workaround.  I had four 
BGP sessions in an admin down state and one that was trying to connect.

I removed all five of these sessions from my configuration and the 
difference was dramatic.

Here's the path/bestpath before the removal of the configuration:

1640389/55395 BGP path/bestpath attribute entries using 262462240 bytes 
of memory

Here it is after:

463745/55388 BGP path/bestpath attribute entries using 74199200 bytes of 
memory

That's a staggering difference.

However, while the memory has been released back into the "BGP memory 
pool", it does not show up in the "free memory pool".  We're still at 
90% usage, so I will have to proceed with our scheduled maintenance tonight.

I had planned on moving to SXI2 tonight, but it sounds like that has 
some memory issues as well.  Think I might just stay put for now, since 
I now know the workaround for this issue.

Thank you everyone for your replies and assistance.  It was of great help!

Cheers!

e ninja wrote:
> Grab multiple captures of sh proc mem to identify the process "holding" 
> and not releasing (i.e. leaking) memory. When memory is heavily 
> depleted, grab a *show memory allocating-process totals* and feel free 
> to unicast.
> 
> Any MALLOC failures?
> 
> -Eninja
> 
> 
> On Sun, Aug 30, 2009 at 10:11 AM, Chris Phillips 
> <cphillips at wbsconnect.com <mailto:cphillips at wbsconnect.com>> wrote:
> 
>     Every six weeks or so I am running out of memory on a 6509 w/ dual
>     SUP720-3BXL with mostly 6700-series line cards.
> 
>     I have 21 other nodes with this exact same configuration, some even
>     running SXI or SXI1 that do not have this issue, which first led me
>     to believe that the issue might be hardware related.
> 
>     During our last maintenance window to alleviate the memory issue, I
>     forced the standby SUP to become the active SUP.  The memory issue
>     persisted, leading me back to thinking it is a software issue.
> 
>     I did not have this issue with SXH* on this same device, but SXH is
>     *SO* buggy, rolling back is not an option.  This leads me to believe
>     that it is most likely a software issue.
> 
>     The router is heavily used with 250+ BGP sessions, OSPF, MPLS,
>     v4/v6, etc, but I don't think it should be consuming and not
>     releasing 4 mbytes of memory each day.
> 
>     Has anyone else seen this?  Anyone know a workaround?
> 
>     I'm upgrading to SXI2 tomight in hopes that it resolves my issue.
> 
>     -- 
>     Chris Phillips
> 
>     _______________________________________________
>     cisco-nsp mailing list  cisco-nsp at puck.nether.net
>     <mailto:cisco-nsp at puck.nether.net>
>     https://puck.nether.net/mailman/listinfo/cisco-nsp
>     archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> 

-- 
Chris Phillips