[c-nsp] VIP/PA question

[Rodney Dunn]

I hate to hear that because usually moving
a bunch of hardware around only makes things
worse.

Can you check the bootflash and see if the
VIP crashinfo files are still there?

There was one nasty bug that we fixed in
12.3(10) that would cause the VIP CPU
to go to 100%.  I'd have to go hunt it
back down.

The main thing to keep in mind is to
make sure the features you run are all
supported in the dCEF path and that
your interfaces are dCEF swithing all
the traffic.

The easiest way to check this is:

sh int stat

Rodney


On Wed, Oct 06, 2004 at 09:07:13AM +0300, Osama wrote:
> No thought or pointers here. I just wanted to say that we went through a 
> similar problem. The VIP CPU utils would hit the roof and it would start 
> dropping packets.
> 
> The throughput of the router was less than 200Mbps total, it had 4 VIPs, 
> a a combination of VIP2s and VIP4s, ATM, FX and Gigabit PAs.
> It started when we upgraded from 12.1(22)E1 to 12.3-5b, to fix some 
> counter problems. Although in our test environment nothing went wrong, 
> one of the VIPs (VIP4-50) starting crashing. So we decided to go back to 
> 12.1(22)E1 until we could figure out why. About 5 days later two VIP CPU 
> utilizations would reach 98-100% and start dropping packets, this 
> happened about 4 times within one day, lasting about 5 minutes each 
> time, then it would go back to forwarding packets. We played around with 
> dCEF and CEF, removing it from some of the cards, placing the load on 
> the RSP. This seemed to calm it down a bit, but on occasion some VIPs 
> would still hit 100%.
> 
> We replaced several VIPs think that it might be a hardware failure, we 
> even replaced the entire chassis and all the interfaces/VIPs, without 
> any success (using the cold standby.)
> 
> Then we decided to reduce the load on the device and we split the 
> traffic between two 7500s.
> Since then we had not had any such incidents, but we still don't know 
> what went wrong.
> Here are the current utils:
> 
> First 7500:
> VIP2-40 CPU utilization for five seconds: 55%/54%; one minute: 54%; five 
> minutes: 55%
> VIP2-50 CPU utilization for five seconds: 2%/2%; one minute: 2%; five 
> minutes: 1%
> VIP4-80 CPU utilization for five seconds: 41%/37%; one minute: 41%; five 
> minutes: 41%
> RSP4    CPU utilization for five seconds: 30%/28%; one minute: 28%; five 
> minutes: 29%
> 
> Second 7500:
> VIP4-50 CPU utilization for five seconds: 27%/27%; one minute: 27%; five 
> minutes: 26%
> VIP2-50 CPU utilization for five seconds: 25%/25%; one minute: 24%; five 
> minutes: 23%
> VIP4-80 CPU utilization for five seconds: 0%/0%; one minute: 0%; five 
> minutes: 0%
> RSP4    CPU utilization for five seconds: 2%/0%; one minute: 2%; five 
> minutes: 2%
> 
> We went through an extremely frustrating time. And were very clueless as 
> to what was happening. Nothing similar existed on mailing list or the 
> web in general. We didn't have TAC support, and were accustomed to 
> solving it our selves.
> 
> I personally detest this box. And very dissatisfied with the quality of 
> this model.
> 
> Sorry I couldn't be of any help. And I feel your pain :-)
> Regards,
> -Osama I. Al-Dosary
> Pete Templin wrote:
> 
> > Oleksandr Pantus wrote:
> >
> >>
> >> Take a look at:
> >>
> >> http://www.cisco.com/warp/public/63/vip_cpu_rxbuffering.html
> >
> >
> > Already familiar with this, and I don't think it applies here.  The 
> > outgoing interfaces are not congested (outbound rates of 40-100kbps on 
> > a T1), yet the router appears to be dropping most or all traffic 
> > headed out those interfaces once the "magic trigger" is pulled.
> >
> > I'm starting to suspect the PA-MC-2T3.  We had an episode where EVERY 
> > T1/frac T1 dropped line protocol (PPP or HDLC) simultaneously.  A week 
> > later, we had the scenario mentioned above, and it's recurred twice 
> > since then.  It's always been the same slot/PA that's been affected, 
> > and the companion PA-FE-TX doesn't appear to be afflicted at all.
> >
> > Any other thoughts?
> >
> > Thanks for the help!
> >
> > Pete
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/