[c-nsp] 3750 stack(2 switches) - High CPU for an hour?

[CiscoNSP List]

Ok - Just an update to this one.

4948 replaced 3750 stack, all appeared "stable" for a couple of days, then OSPF up/downs started again....but only on the portchan subints....so, moved one of our Inter-pop links(That was impacted+using portchan subint) from connecting to 4948, to directly into a port on the ASR1006 (This is the L3 device the portchan terminates on)....OSPF up/downs continued, but not for this interpop link....so we thought it had to be "something" to do with the the portchan.

To test this theory, we moved the interpop link back to the 4948, and created a dedicated trunk link from 4948->ASR1006 for this link(New ports on both 4948+ASR1006, so not using the portchan at all)...OSPF up/downs continued for the portchan subint services, but also now re-started on the interpop link....So it seems that "any" link that uses this switch is seeing random OSPF+MPLS up/downs (At present, they are lasting ~3-5seconds).

All the Interfaces are clean (No drops/errors), the only "thing" that seems to coincide with the drops is that the CPU on the 4948 does spike(~70+%)

Im going to open a TAC case, but any suggestions on what could be causing this are greatly appreciated!
   


> From: cisconsp_list at hotmail.com
> To: adam1984 at gmail.com
> Date: Thu, 4 Dec 2014 06:10:16 +1100
> CC: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] 3750 stack(2 switches) - High CPU for an hour?
> 
> Thanks for the reply - Yes, we have also been hit with snmp/high CPU...and we have storm control on cust ports.
> 
> It coincided with a customer making some changes to there network(That connects to this switch stack) just before the high cpu issue, and resolved itself...but as I said, nothing logged (It is also configured to send snmp traps for high CPU, but no traps were sent)....anyway, 2 days of stability, then it occurred again yesterday(Only much more briefly(30seconds)....again, nothing logged, no traps...it then occurred again ~1hour later, same symptoms. I tried creating a test vlan(As it is also configured to send trap if vlan is created/deleted) - This also caused high CPU(Nothing logged, no traps, and some OSPF adjaceny up/downs)....So, did an emergency replacement of the switch(Swapped it out with a 4948).
> 
> 
> 
> Date: Tue, 2 Dec 2014 10:27:53 +1000
> Subject: Re: [c-nsp] 3750 stack(2 switches) - High CPU for an hour?
> From: adam1984 at gmail.com
> To: cisconsp_list at hotmail.com
> CC: cisco-nsp at puck.nether.net
> 
> We have seen high CPU Caused by SNMP.. we have implemented a view that helped. viewing the flash and license mib causes problems.
> 
> snmp-server view protect iso included
> snmp-server view protect ciscoLicenseMgmtMIB excluded
> snmp-server view protect ciscoFlashFileEntry excluded
> 
> snmp-server community public view protect RO
> 
> also, it might be worth implementing storm control on the client facing switchports.
> 
> On 1 December 2014 at 19:21, CiscoNSP List <cisconsp_list at hotmail.com> wrote:
> Hi Everyone,
> 
> 
> 
> Strange situation on the weekend...started losing OSPF adjacencies to devices connected to(through) a 3750 stack (The 3750 is only doing L2, so routers connected together via the 3750)...It lasted ~1 hour (Happened while I was on a plane, so the only info I have are logs and snmp graphs)....CPU hit ~85%, traffic on all ports was "normal", PPS was "normal", switch didnt reboot and nothing at all in the logs....If the switch got hit with a mac flood or loop, I would have thought it would have a logged "something"?
> 
> 
> 
> Any suggestions on what else could have caused this?
> 
> 
> 
> Cheers.
> 
> _______________________________________________
> 
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> 
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> 
> archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> 
>  		 	   		  
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/