[c-nsp] Two DS3s Bounced at the Same Time

[Rodney Dunn]

To see the error history do:

Router#sh controllers t3 3/0/0 err
  T3 3/0/0: Error Log Information
  present alarm: LOS
  Since 13:04:09
  T1 1 Error Log Information
  present alarm: NONE

and at the T1 level:

Router#sh controllers t3 3/0/0 /1 err
  T3 3/0/0: Error Log Information
  present alarm: LOS
  Since 13:04:09
  T1 1 Error Log Information
  present alarm: NONE
Router#


On Sun, Mar 20, 2005 at 09:09:34AM -0500, Rodney Dunn wrote:
> John,
> 
> I worked on an issue a couple months ago and we found that
> a couple of our T3 PA's were nto compliant with the ANSI
> specification in regards to the soak period for reacting
> to an alarm.
> 
> We were reacting immediately and you would see the controller
> flap.
> 
> You may want to check out:
> 
> CSCee70591
> Externally found severe defect: Resolved (R)
> PA-2T3+ does not adhere to the ANSI T1.231 standard
> 
> and for the MC PA it's:
> 
> CSCee49862
> Externally found severe defect: Resolved (R)
> PA-MC-2T3+ does not adhere to ANSI T1.231 standard
> 
> we added some really nice (I think) alarm history capability under:
> 
> CSCee49983
> Externally found severe defect: Resolved (R)
> PA-MC-2T3+ should keep a history of most recent alarms
> 
> to help you look at the history after a hit and see what type of
> alarm the controller saw.
> 
> Rodney
> 
> 
> On Sat, Mar 19, 2005 at 05:06:51PM -0700, John Neiberger wrote:
> > We have an OC-12 coming into our main facility, and are using four
> > DS3s on that OC12 at the moment. Two of the DS3s are channelized and
> > mostly carry voice circuits. Two of the DS3s are not channelized and
> > carry data. The two data DS3s terminate on two PA-T3+ port adapters
> > that reside on the same VIP2-50.
> > 
> > On Friday, in the middle of the day, both DS3s went down at precisely
> > the same second, but they came back up within a minute. This still
> > caused some rather nasty network issues because of the visibility of
> > the traffic on those links. The problem is that we can't find a reason
> > for both of them to go down.
> > 
> > This could have been a problem on the VIP or the port adapters but
> > there were no error messages in the logs other than the link down
> > messages. The OC12 did not bounce because our two channelized DS3s
> > stayed up. Qwest has verified that they saw our two data DS3s go down
> > and they thought they some some errors in one of the relevant central
> > offices. However, after intrusive testing last night they were not
> > able to find a problem.
> > 
> > Needless to say, my CTO and VP over our department are not happy that
> > this has happened and we can't find any reason for it. I'm wondering
> > if the problem actually is on the router but I don't know what to look
> > for when there are no error messages in the logs.
> > 
> > The CPU usage on the router was maxed for at least a few seconds, but
> > not more than a minute. BGP is running on both links but they are not
> > Internet connection and there are only about 700 prefixes on each
> > link. I see no alignment errors, and I have plenty of memory on the
> > RSP. The links have been running cleanly all day today and my BGP
> > peers have remained stable.
> > 
> > Any thoughts? I don't even really know what else to look for. I still
> > have a few questions for Qwest but I am starting to think perhaps my
> > VIP freaked out for a moment.
> > 
> > Thanks,
> > John
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/