[c-nsp] VIP/PA question

[Osama]

No thought or pointers here. I just wanted to say that we went through a 
similar problem. The VIP CPU utils would hit the roof and it would start 
dropping packets.

The throughput of the router was less than 200Mbps total, it had 4 VIPs, 
a a combination of VIP2s and VIP4s, ATM, FX and Gigabit PAs.
It started when we upgraded from 12.1(22)E1 to 12.3-5b, to fix some 
counter problems. Although in our test environment nothing went wrong, 
one of the VIPs (VIP4-50) starting crashing. So we decided to go back to 
12.1(22)E1 until we could figure out why. About 5 days later two VIP CPU 
utilizations would reach 98-100% and start dropping packets, this 
happened about 4 times within one day, lasting about 5 minutes each 
time, then it would go back to forwarding packets. We played around with 
dCEF and CEF, removing it from some of the cards, placing the load on 
the RSP. This seemed to calm it down a bit, but on occasion some VIPs 
would still hit 100%.

We replaced several VIPs think that it might be a hardware failure, we 
even replaced the entire chassis and all the interfaces/VIPs, without 
any success (using the cold standby.)

Then we decided to reduce the load on the device and we split the 
traffic between two 7500s.
Since then we had not had any such incidents, but we still don't know 
what went wrong.
Here are the current utils:

First 7500:
VIP2-40 CPU utilization for five seconds: 55%/54%; one minute: 54%; five 
minutes: 55%
VIP2-50 CPU utilization for five seconds: 2%/2%; one minute: 2%; five 
minutes: 1%
VIP4-80 CPU utilization for five seconds: 41%/37%; one minute: 41%; five 
minutes: 41%
RSP4    CPU utilization for five seconds: 30%/28%; one minute: 28%; five 
minutes: 29%

Second 7500:
VIP4-50 CPU utilization for five seconds: 27%/27%; one minute: 27%; five 
minutes: 26%
VIP2-50 CPU utilization for five seconds: 25%/25%; one minute: 24%; five 
minutes: 23%
VIP4-80 CPU utilization for five seconds: 0%/0%; one minute: 0%; five 
minutes: 0%
RSP4    CPU utilization for five seconds: 2%/0%; one minute: 2%; five 
minutes: 2%

We went through an extremely frustrating time. And were very clueless as 
to what was happening. Nothing similar existed on mailing list or the 
web in general. We didn't have TAC support, and were accustomed to 
solving it our selves.

I personally detest this box. And very dissatisfied with the quality of 
this model.

Sorry I couldn't be of any help. And I feel your pain :-)
Regards,
-Osama I. Al-Dosary
Pete Templin wrote:

> Oleksandr Pantus wrote:
>
>>
>> Take a look at:
>>
>> http://www.cisco.com/warp/public/63/vip_cpu_rxbuffering.html
>
>
> Already familiar with this, and I don't think it applies here.  The 
> outgoing interfaces are not congested (outbound rates of 40-100kbps on 
> a T1), yet the router appears to be dropping most or all traffic 
> headed out those interfaces once the "magic trigger" is pulled.
>
> I'm starting to suspect the PA-MC-2T3.  We had an episode where EVERY 
> T1/frac T1 dropped line protocol (PPP or HDLC) simultaneously.  A week 
> later, we had the scenario mentioned above, and it's recurred twice 
> since then.  It's always been the same slot/PA that's been affected, 
> and the companion PA-FE-TX doesn't appear to be afflicted at all.
>
> Any other thoughts?
>
> Thanks for the help!
>
> Pete
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/