[c-nsp] Two DS3s Bounced at the Same Time

[Bill Wichers]

> This could have been a problem on the VIP or the port adapters but
> there were no error messages in the logs other than the link down
> messages. The OC12 did not bounce because our two channelized DS3s
> stayed up. Qwest has verified that they saw our two data DS3s go down
> and they thought they some some errors in one of the relevant central
> offices. However, after intrusive testing last night they were not
> able to find a problem.
[snip]
> The CPU usage on the router was maxed for at least a few seconds, but
> not more than a minute. BGP is running on both links but they are not
> Internet connection and there are only about 700 prefixes on each
> link. I see no alignment errors, and I have plenty of memory on the
> RSP. The links have been running cleanly all day today and my BGP
> peers have remained stable.
>
> Any thoughts? I don't even really know what else to look for. I still
> have a few questions for Qwest but I am starting to think perhaps my
> VIP freaked out for a moment.

I've seen VIP2-50's do strange things when hit with DoS attacks -- even
relatively small ones. What usually would happen is the VIP2 is in the
path of the DoS traffic, which maxes it's CPU and causes the BGP session
to drop (from the filled pipe and/or saturated CPU). The traffic levels to
do this sometimes are really small -- 20 Mb/s or so -- but consist of
minimum size packets to random destinations. I've seen this happen maybe
twice on a DS3 we have between to POPs.

Although unlikely if both circuits went out at the same time, you might
want to check the BNC connectors on the DS3 cabling. I've seen DS3 cables
that get snagged and pulled a bit bend the center pins (usually the prongs
for the center pin on the jack get bent), which results in an unreliable
connection.

BTW, on my system the `show cont t3 ...` command only works on
*channelized* T3 interfaces, not PODS3 interfaces. You could try `show
cont serail <interface number>` instead, although it doesn't have the same
level of detail at the circuit level.

     -Bill


*****************************
Waveform Technology
UNIX Systems Administrator