[c-nsp] Two DS3s Bounced at the Same Time

[Rodney Dunn]

John,

I worked on an issue a couple months ago and we found that
a couple of our T3 PA's were nto compliant with the ANSI
specification in regards to the soak period for reacting
to an alarm.

We were reacting immediately and you would see the controller
flap.

You may want to check out:

CSCee70591
Externally found severe defect: Resolved (R)
PA-2T3+ does not adhere to the ANSI T1.231 standard

and for the MC PA it's:

CSCee49862
Externally found severe defect: Resolved (R)
PA-MC-2T3+ does not adhere to ANSI T1.231 standard

we added some really nice (I think) alarm history capability under:

CSCee49983
Externally found severe defect: Resolved (R)
PA-MC-2T3+ should keep a history of most recent alarms

to help you look at the history after a hit and see what type of
alarm the controller saw.

Rodney


On Sat, Mar 19, 2005 at 05:06:51PM -0700, John Neiberger wrote:
> We have an OC-12 coming into our main facility, and are using four
> DS3s on that OC12 at the moment. Two of the DS3s are channelized and
> mostly carry voice circuits. Two of the DS3s are not channelized and
> carry data. The two data DS3s terminate on two PA-T3+ port adapters
> that reside on the same VIP2-50.
> 
> On Friday, in the middle of the day, both DS3s went down at precisely
> the same second, but they came back up within a minute. This still
> caused some rather nasty network issues because of the visibility of
> the traffic on those links. The problem is that we can't find a reason
> for both of them to go down.
> 
> This could have been a problem on the VIP or the port adapters but
> there were no error messages in the logs other than the link down
> messages. The OC12 did not bounce because our two channelized DS3s
> stayed up. Qwest has verified that they saw our two data DS3s go down
> and they thought they some some errors in one of the relevant central
> offices. However, after intrusive testing last night they were not
> able to find a problem.
> 
> Needless to say, my CTO and VP over our department are not happy that
> this has happened and we can't find any reason for it. I'm wondering
> if the problem actually is on the router but I don't know what to look
> for when there are no error messages in the logs.
> 
> The CPU usage on the router was maxed for at least a few seconds, but
> not more than a minute. BGP is running on both links but they are not
> Internet connection and there are only about 700 prefixes on each
> link. I see no alignment errors, and I have plenty of memory on the
> RSP. The links have been running cleanly all day today and my BGP
> peers have remained stable.
> 
> Any thoughts? I don't even really know what else to look for. I still
> have a few questions for Qwest but I am starting to think perhaps my
> VIP freaked out for a moment.
> 
> Thanks,
> John
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/