[c-nsp] 3550 IO memory fragmentation

[Matlock, Kenneth L]

Do you have traffic graphs during this timeframe? Maybe a DDOS at or
through these boxes tied up the available memory. Especially since 'I/O'
was the pool it was trying to grab from at the time?

Ken Matlock
Network Analyst
Exempla Healthcare
(303) 467-4671
matlockk at exempla.org


-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Jon Lewis
Sent: Monday, January 04, 2010 2:01 PM
To: David Freedman
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] 3550 IO memory fragmentation

Most of the 3550s, including the one that crashed, are running 
121-22.EA10b.  CSCdz51522 seems unlikely as there was nobody logged in 
making changes and nobody should have been making (and there are no
logged 
signs of) physical changes to the network at the time of the event.

On Mon, 4 Jan 2010, David Freedman wrote:

> What release are you running? could it be CSCdz51522?
>
> Dave.
>
>
> Jon Lewis wrote:
>> We had a recent network event during which all of our 3550 access
layer
>> switches started logging things like:
>>
>> %SYS-2-MALLOCFAIL: Memory allocation of 1680 bytes failed from
0x17FC48,
>> alignment 0
>> Pool: I/O  Free: 5936  Cause: Memory fragmentation
>> Alternate Pool: None  Free: 0  Cause: No Alternate pool
>>
>> -Process= "Pool Manager", ipl= 0, pid= 5
>> -Traceback= 1C919C 1CA760 17FC4C 1DB43C 1DB658 1EBACC 1EF0FC
>>
>> A few minutes later, all was back to normal, though one 3550 did a
>> software forced crash / reload.
>>
>> Under normal circumstances, these switches have 4.5-5mb of free IO
memory.
>>
>> This looks very similar to what was posted several years ago at
>>
>> http://www.velocityreviews.com/forums/t31947-catalysts-trouble.html
>>
>> Searching bug toolkit, I didn't find anything that looked relevant.
Has
>> anyone else run into this sort of thing with 12.1EA software or have
an
>> idea what the cause/solutions might be?
>>
>>
----------------------------------------------------------------------
>>  Jon Lewis                   |  I route
>>  Senior Network Engineer     |  therefore you are
>>  Atlantic Net                |
>> _________ http://www.lewis.org/~jlewis/pgp for PGP public
key_________
>> _______________________________________________
>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>

----------------------------------------------------------------------
  Jon Lewis                   |  I route
  Senior Network Engineer     |  therefore you are
  Atlantic Net                |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/