[c-nsp] Two DS3s Bounced at the Same Time

[John Neiberger]

We have an OC-12 coming into our main facility, and are using four
DS3s on that OC12 at the moment. Two of the DS3s are channelized and
mostly carry voice circuits. Two of the DS3s are not channelized and
carry data. The two data DS3s terminate on two PA-T3+ port adapters
that reside on the same VIP2-50.

On Friday, in the middle of the day, both DS3s went down at precisely
the same second, but they came back up within a minute. This still
caused some rather nasty network issues because of the visibility of
the traffic on those links. The problem is that we can't find a reason
for both of them to go down.

This could have been a problem on the VIP or the port adapters but
there were no error messages in the logs other than the link down
messages. The OC12 did not bounce because our two channelized DS3s
stayed up. Qwest has verified that they saw our two data DS3s go down
and they thought they some some errors in one of the relevant central
offices. However, after intrusive testing last night they were not
able to find a problem.

Needless to say, my CTO and VP over our department are not happy that
this has happened and we can't find any reason for it. I'm wondering
if the problem actually is on the router but I don't know what to look
for when there are no error messages in the logs.

The CPU usage on the router was maxed for at least a few seconds, but
not more than a minute. BGP is running on both links but they are not
Internet connection and there are only about 700 prefixes on each
link. I see no alignment errors, and I have plenty of memory on the
RSP. The links have been running cleanly all day today and my BGP
peers have remained stable.

Any thoughts? I don't even really know what else to look for. I still
have a few questions for Qwest but I am starting to think perhaps my
VIP freaked out for a moment.

Thanks,
John