[c-nsp] Debugging T1 Bounces on CT3

[Greg Boehnlein]

Hello all,
	We have a 7507 w/ RSP4/256M and a VIP-2-50 w/ CT3 card running 
rsp-pv-mz.123-11.T3.bin. The CT3 card is plugged into an Adtran MX2800 DS3 
mux. The Adtran is used to consolidate DS1 circuits from various carriers 
into a single Channelized DS3. So, the layout looks like this:

Carrier A -> DS1 -> MX2800 -> CT3 (7507)
Carrier B -> DS1 ---^

For the most part, this has been flawless and very stable up until about a 
week ago. Our logs show constant T1 controller failures as far back as 
3/26/05 and continuing up to the present.

On the Cisco, the DS3 span looks very clean:

  CT3 H/W Version: 1.0.1, CT3 ROM Version: 1.1, CT3 F/W Version: 2.4.3
  FREEDM version: 1, reset 0 resurrect 0
  Applique type is Channelized T3
  Description: Muxed DS3 to Adtran MX2800
  No alarms detected.
  Framing is M23, Line Code is B3ZS, Clock Source is Line
  Rx-error throttling on T1's ENABLED
  Rx throttle total 306, equipment customer loopback
  Data in current interval (453 seconds elapsed):
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation, 0 P-bit Err Secs
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs
     0 Unavailable Secs, 0 Line Errored Secs
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs
  Total Data (last 24 hours)
     0 Line Code Violations, 0 P-bit Coding Violation,
     0 C-bit Coding Violation, 0 P-bit Err Secs,
     0 P-bit Severely Err Secs, 0 Severely Err Framing Secs,
     0 Unavailable Secs, 0 Line Errored Secs,
     0 C-bit Errored Secs, 0 C-bit Severely Errored Secs

On the MX2800, however we are showing the following errors at the DS3 
level in the last 24 hours:

  Unavailable Seconds (UAS)                   279
  Severely Errored Framing Sec (SEFS)           0
  Line Coding Violations (LCV)                  9
  Line Errored Seconds (LES)                  334
  Line Severely Errored Seconds (LSES)        333
  P-Bit Errored Seconds (PES)                  53
  P-Bit Severely Errored Sec (PSES)            49
  P-Bit Coding Violations (PCV)                62
  C-Bit Coding Violations (CCV)                 0
  C-Bit Errored Seconds (CES)                   0
  C-Bit Severely Errored Sec (CSES)             0
  F-Bit Errors (FBE)                        77576
  Far End Block Error (FEBE)                    0

As a result, the DS1 spans on the Cisco look like complete crap, and we 
have the following in our logs:

 Apr  4 20:12:59: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to DOWN
 Apr  4 20:13:01: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to UP
 Apr  4 20:13:04: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 11, changed state to DOWN
 Apr  4 20:13:21: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 11, changed state to UP
 Apr  4 20:23:15: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to DOWN
 Apr  4 20:23:17: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to UP
 Apr  4 20:46:40: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to DOWN
 Apr  4 20:46:42: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 6, changed state to DOWN
 Apr  4 20:46:42: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 16, changed state to UP
 Apr  4 20:47:00: %CONTROLLER-5-UPDOWN: Controller T3 1/0/0 T1 6, changed state to UP

Just an example of the Cisco's T1 stats:

  T1 2 is up
  timeslots: 1-24
  FDL per AT&T 54016 spec.
  No alarms detected.
  Framing is ESF, Clock Source is Line, Rx throttles 1
  Data in current interval (79 seconds elapsed):
     0 Line Code Violations, 0 Path Code Violations
     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs
     0 Unavail Secs, 0 Stuffed Secs
  Total Data (last 24 hours)
     0 Line Code Violations,2284 Path Code Violations,
     0 Slip Secs, 3 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,
     5 Errored Secs, 2 Bursty Err Secs, 3 Severely Err Secs
     260 Unavail Secs, 0 Stuffed Secs

Any suggestions on what I should look at? I'm tempted to swap out the 
cable between the MUX and the Router... 

-- 
    Vice President of N2Net, a New Age Consulting Service, Inc. Company
         http://www.n2net.net Where everything clicks into place!
                             KP-216-121-ST