[c-nsp] Very Strange Problems with 7513

[John Neiberger]

Earlier in the year I had a 7513 that began to see some strange issues.
It started with a VIP (w/two DS3 PAs) crashing. It recovered and we
didn't think much of it until a few weeks later when another VIP (w/ATM
IMA PA) crashed and did not recover. We replaced the VIP and the ATM PA
and thought all was well. Within a short span of time, the RSP began to
fail with parity errors. TAC told me that this was usually caused by
cosmic rays and that I didn't need to worry about it unless it happened
again. Well, it happened again and we replaced the RSP. We eventually
ended up replacing the VIP and ATM PA *again*, the RSP *again*, and the
other VIP with the DS3 PAs, culminating in replacing the chassis.

That's the short story. Fast forward to October. Different cards
(mostly, except for two CIPs, two FSIPs, and some old Ethernet cards)
and different chassis. The VIP with the two DS3 PAs reloaded but
recovered. TAC told me that is was a soft parity error and that I didn't
need to worry about it unless it happened again. Nothing happened for
two weeks and then the RSP crashed. We replaced it and all has been well
until two days ago when the VIP with the ATM PA died. I came in last
night and replaced it, and my RSP crashed! Again, it is with memory
parity errors, and it crashed three times in just a few hours. I
replaced it with a spare RSP so I think everything is stable for the
moment.

But, TAC appears to be stumped. The crash files for the VIPs and the
RSP make it look as if they are entirely unrelated events, but they most
certainly are related somehow...but how? This is a new chassis and
mostly new cards. I have two thoughts: 1) a bad power feed from the PDU
in the data center, or 2) one or more of the old cards that were moved
from the old chassis are failing in such a way as to appear to be
functioning correctly yet cause problems with other cards. 

Any thoughts?

Thanks,
John
--