[c-nsp] Malloc fail on 7206VXR

[Rodney Dunn]

I'm responding back to the alias to close the loop
on this issue.

Oleksandr was kind enough to get me some data to
look at this problem.

In looking at the full 'sh mem' dump for the blocks
that are in use and the ones are not I noticed a
pattern of blocks:

          65536    Free list 26
65C37A20 0000065544 65C279F4 65C47A60 000  0        66053D10 60757398  (coalesced)

of size 65544.

This is a bug in the way these blocks were being handled.

The fix for this bug is:

CSCuk51673
Externally found catastrophic defect: Resolved (R)
malloc_aligned adds unnecessary padding

and it's being put in the 12.2(18)S and 12.2(20)S
throttle branches.

A possible workaround to this problem in the interim
is to configure a free-list pool size of:

memory free-list 65544
 
so those blocks would be reused.

12.2(25)S code or later does not have this problem because
of added memory enhancement feature.

Rodney


On Thu, Oct 21, 2004 at 09:41:13PM -0400, Rodney Dunn wrote:
> It sounds like a lot of small blocks were used and
> freed back.  But what is strange there is the Lowest
> column should be lower if the memory was actually
> used.  So in your case it's as if a few small blocks
> were used over and over to cause the fragmentation.
> 
> It would be interesting to see:
> sh ver
> sh mem free
> 
> Unfortunately though unless you are on 12.2(20.04)S or
> later if the blocks are listed as "fragments" we have
> no way to tell who the original memory requestor was that
> caused a fragment to exist.  I had DE add an update
> so we could have some history of the block and that
> code went in 12.2(20.4)S.
> 
> I want to say it was most likely BGP transient memory usage
> but if that were the case I would have expected the
> "lowest" mark to have been lower but that wasn't the case.
> 
> I took a quick glance at all the bugs that went in
> between 18S3 and 18S5 and the only one that jumped out
> at me was a cef/dCEF toggling issue that resulted in
> a memory leak.  That wouldn't apply here.
> 
> Did you reboot the box to clean it up?
> 
> Rodney
> 
> 
> On Mon, Oct 11, 2004 at 02:12:46PM +0300, Oleksandr Pantus wrote:
> > Hello !
> > 
> > Have a trouble with 7206VXR NPE-300. There was a flap of BGP session
> > with our main upstream, and router got fullview BGP table from another border
> > router (back-up link) by iBGP. Main upstream came up, and router relearned
> > fullview from it. After such a maneuvres, I start to get following
> > error messages almost every minute:
> > 
> > Oct 11 13:54:25.731: %SYS-2-MALLOCFAIL: Memory allocation of 262152 bytes failed from 0x60737F04, alignment 32 
> > Pool: Processor  Free: 45345896  Cause: Memory fragmentation 
> > Alternate Pool: None  Free: 0  Cause: No Alternate pool 
> > 
> > -Process= "Per-Second Jobs", ipl= 3, pid= 28
> > -Traceback= 60733674 60737294 60737F0C 605247E8 60524A6C 605286F0 607445D4 606BE028
> > 
> > Here is "shov version" output.
> > 
> > Cisco Internetwork Operating System Software 
> > IOS (tm) 7200 Software (C7200-P-M), Version 12.2(18)S5,  RELEASE SOFTWARE (fc1)
> > 
> > cisco 7206VXR (NPE300) processor (revision B) with 229376K/65536K bytes 
> > of memory.
> > Processor board ID 13253888
> > R7000 CPU at 262Mhz, Implementation 39, Rev 1.0, 256KB L2, 2048KB L3 Cache
> > 6 slot VXR midplane, Version 2.0
> > 
> > And "show memory free"/
> > 
> >                Head    Total(b)     Used(b)    Free(b)   Lowest(b)  Largest(b)
> > Processor  6240DCA0   197075808   151750648   45325160   45177992     131144
> >       I/O  20000000    33554432      906592   32647840   32646288   30496312
> >     I/O-2   E000000    33554456     4895528   28658928   28658928   28658744
> > 
> > 
> > It seems that 12.2(18)S5 have a memory leak due to fragmentation.
> > There was no such a problem with 12.2(18)S3. 
> > 
> > Or there is some other explanation to the small size of max
> > allocated block while there is about 45M of free processor memory ?
> > 
> > -- 
> > S/Y,
> > Alexander, MD, 			nic-hdl: AJP1-UANIC
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/