[c-nsp] DS3 problem on PA-T3

[Jay Nakamura]

I am stumped on a problem with our Telco here.

We have a frame-relay DS3 from Verizon.  This is how it's connected from the
CO.

Verizon CO Frame router/switch <-> DS3 <-> Adtran Opti3 <-> OC3 fiber <->
Adtran Opti3 <-> DS3 <-> our router

We started having problem two weeks ago and the circuit started bouncing.
We originally had a 7500 router there.  Verizon kept insisting the problem
is CPE so we have tried second DS3 interface on the 7500, different card in
the 7500, then finally swapping out the entire router with a 7200 and
PA-T3.  Verizon finally figured out there was indeed something wrong on
their side and swapped out some cards on the Opti3.  Now the circuit isn't
going up and down but we are seeing trickling errors on the interface
consistantly.  They put a T-berd on our side of the dmark and monitored the
circuit over the weekend.  They are seeing errors coming from our router and
FEBE from CO end.  From our router, I am seeing errors coming from Verizon.

At this point, I can't believe it's our DS3 card.  We have changed the DS3
cable as well.  Does anyone see any possible problem on my side of the
config?  Or any other ideas what could be causing this problem?

interface Serial1/0
 description Verizon DS3 frame-relay
 no ip address
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 encapsulation frame-relay IETF
 no ip mroute-cache
 logging event subif-link-status
 logging event dlci-status-change
 framing c-bit
 cablelength 50
 dsu bandwidth 44210
 serial restart-delay 0
 frame-relay lmi-type ansi

#sh controllers ser 1/0
M1T-T3 pa: show controller:
PAS unit 0, subunit 0, f/w version 3-92, rev ID 0x2800001, version 3
idb = 0x614754AC, ds = 0x61476FC4, ssb=0x614772F4
Clock mux=0x30, ucmd_ctrl=0x0, port_status=0x1
Serial config=0x8, line config=0x1B0202
maxdgram=4584, bufpool=128Kb, 256 particles

   rxLOS inactive, rxLOF inactive, rxAIS inactive
   txAIS inactive, rxRAI inactive, txRAI inactive
line state: up
cable type : T3  cable, received clockrate 44201430

base0 registers=0x3C800000, base1 registers=0x3C802000
rx_base0 registers=0x3C804000, rx_base1 registers=0x3C806000
mxt_ds=0x61922350, rx ring entries=124, tx ring entries=254
statring=0x3903900, statr shadow=0x61479124, stat_head=175
plus_pa: rx_statring=0x3903D40, rx_statr shadow=0x61479950, rx_stat_head=29
rxring=0x39036C0, rxr shadow=0x61478AF8, rx_head=109
txring=0x3904300, txr shadow=0x61479D7C, tx_head=185, tx_tail=191,
tx_count=6
throttled=0, enabled=0
halted=0, last halt reason=0
Microcode fatal errors=0
rx_no_eop_err=0, rx_no_stp_err=0, rx_no_eop_stp_err=0
rx_no_buf=0, rx_soft_overrun_err=23, dump_err= 0, bogus=0, mxt_flags=0x2C
tx_underrun_err=18, tx_soft_underrun_err=0, tx_limited=0(254)
tx_fullring=538, tx_started=809952433
rx_int_count=870161881, tx_int_count=1611065176
   Framing is c-bit, Clock Source is Line
   Bandwidth limit is 44210, DSU mode 0, Cable length is 50
   rx FEBE since last clear counter 1980, since reset 179
   Data in current interval (881 seconds elapsed):
     1 Line Code Violations, 1 P-bit Coding Violation
     1 C-bit Coding Violation
     1 P-bit Err Secs, 0 P-bit Sev Err Secs
     0 Sev Err Framing Secs, 0 Unavailable Secs
     1 Line Errored Secs, 1 C-bit Errored Secs, 0 C-bit Sev Err Secs
   Total Data (last 24 hours)
     2343 Line Code Violations, 637 P-bit Coding Violation,
     635 C-bit Coding Violation,
     260 P-bit Err Secs, 0 P-bit Sev Err Secs,
     0 Sev Err Framing Secs, 0 Unavailable Secs,
     418 Line Errored Secs, 260 C-bit Errored Secs, 0 C-bit Sev Err Secs

   No alarms detected.

#sh int ser 1/0
Serial1/0 is up, line protocol is up
  Hardware is M1T-T3 pa
  Description: Verizon DS3 frame-relay #########
  MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
     reliability 255/255, txload 57/255, rxload 13/255
  Encapsulation FRAME-RELAY IETF, crc 16, loopback not set
  Keepalive set (10 sec)
  Restart-Delay is 0 secs
  LMI enq sent  41313, LMI stat recvd 41313, LMI upd recvd 0, DTE LMI up
  LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0
  LMI DLCI 0  LMI type is ANSI Annex D  frame relay DTE
  Broadcast queue 0/256, broadcasts sent/dropped 0/0, interface broadcasts 0
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 4d18h
  Input queue: 0/75/0/3084 (size/max/drops/flushes); Total output drops:
4570
  Queueing strategy: fifo
  Output queue :0/40 (size/max)
  5 minute input rate 2283000 bits/sec, 1030 packets/sec
  5 minute output rate 10030000 bits/sec, 1403 packets/sec
     209774076 packets input, 1379501582 bytes, 0 no buffer
     Received 356689 broadcasts, 0 runts, 0 giants, 0 throttles
              0 parity
     4829 input errors, 4748 CRC, 0 frame, 79 overrun, 0 ignored, 2 abort
     259718552 packets output, 3599585926 bytes, 0 underruns
     0 output errors, 0 applique, 4294967295 interface resets
     0 output buffer failures, 0 output buffers swapped out
     1 carrier transitions
   rxLOS inactive, rxLOF inactive, rxAIS inactive
   txAIS inactive, rxRAI inactive, txRAI inactive