Thanks to everyone who responded. It was in fact not a configuration or
hardware problem with us.
The problem was isolated to an OC48 trunk card (recieve) in between two POPs
(going CBIT<-->M13) that was freaking out every so often. The carrier had
other tickets opened by other customers complaining of similar issues, but
was able to convince these customers it was someone else's fault. Only took
an on-site techs about 47 hrs of combined time to isolate it.
Now they'll still believe it because it was fixed without their
intervention. :)
Whatever happened to tier-2 groups that monitored core backbone components?
Regards,
Deepak Jain
AiNET
-----Original Message-----
From: Deepak Jain [mailto:deepak@ai.net]
Sent: Saturday, May 26, 2001 5:36 AM
To: Cisco-Nsp@Puck. Nether. Net
Subject: [nsp] DS3 strangeness
I've got a pair of DS3s that are acting up.
The topology is like this:
|-------------DS3 1-------|
R1 |-------------DS3 2-------| R2
For months they have been perfect. In the last 24 hrs, side R1 has started
clocking _many_ carrier transitions. A fraction of that number are visible
on R2. These peak at 100% load (only new change in the last few days). Both
R1 and R2 are Cisco routers, both are terminating the pair of DS3s onto
PA-2T3+'s.
I could have hardware start being swapped, but I haven't decided whether to
start on the side seeing the most errors or the least. In the 15 minutes, R1
has seen 20 transitions (equivalent numbers on both circuits) and R2 has
seen none.
In the handful of minutes it took to write this email, R1's transitions have
jumped to 40, and R2 sees none. R2 sees "long" ones -- ones that last more
than a few seconds.
The DS3s are provided over most of the same equipment, and have sequential
circuit IDs, but there is some diversity in their engineering.
Both sides are reporting Unavailable Seconds, but no other errors. There are
no reasons in the log for why the carrier is being transitioned. It does not
appear to happen on each circuit at the exact moment, but similar numbers
emerge after extended run-times.
I'd appreciate any debugging suggestions -- Thanks!
Deepak Jain
AiNET
R1's side
Serial1/0/0 is up, line protocol is up
Hardware is cyBus PODS3+ Serial
Internet address is 205.134.160.21/30
MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec, rely 255/255, load 148/255
Encapsulation HDLC, loopback not set, keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:13:05
Queueing strategy: fifo
Output queue 0/1024, 0 drops; input queue 0/75, 0 drops
5 minute input rate 2878000 bits/sec, 3976 packets/sec
5 minute output rate 25778000 bits/sec, 4705 packets/sec
3200215 packets input, 295602574 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
5384 input errors, 4223 CRC, 0 frame, 1040 overrun, 0 ignored, 121
abort
3782478 packets output, 2601341208 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffers copied, 0 interrupts, 0 failures
20 carrier transitions
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
Serial1/0/1 is up, line protocol is Serial1/0/0 -
Mx T3+(2) HW Revision 0x3, FW Revision 3.9
Framing is m13, Clock Source is Line
Bandwidth limit is 44210, DSU mode 0, Cable length is 10
Scrambling is enabled
Data in current interval (182 seconds elapsed):
0 Line Code Violations, 0 P-bit Coding Violation
0 C-bit Coding Violation
0 P-bit Err Secs, 0 P-bit Sev Err Secs
0 Sev Err Framing Secs, 27 Unavailable Secs
0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
Total Data (last 1 15 minute intervals):
0 Line Code Violations, 0 P-bit Coding Violation,
0 C-bit Coding Violation,
0 P-bit Err Secs, 0 P-bit Sev Err Secs,
0 Sev Err Framing Secs, 17 Unavailable Secs,
0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
No alarms detected.
Tabular MIB :
INTERVAL LCV PCV CCV PES PSES SEFS UAS LES CES CSES
05:21-05:24 0 0 0 0 0 0 27 0 0 0
05:06-05:21 0 0 0 0 0 0 17 0 0 0up
Hardware is cyBus PODS3+ Serial
Description: T3 to Level 3 (UAA-0002)
Internet address is 205.134.160.25/30
MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec, rely 255/255, load 134/255
Encapsulation HDLC, loopback not set, keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:14:41
Queueing strategy: fifo
Output queue 0/1024, 0 drops; input queue 0/75, 0 drops
5 minute input rate 3075000 bits/sec, 3979 packets/sec
5 minute output rate 23241000 bits/sec, 4349 packets/sec
3506257 packets input, 336335955 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
5624 input errors, 4178 CRC, 0 frame, 1332 overrun, 0 ignored, 114
abort
3835894 packets output, 2595870843 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffers copied, 0 interrupts, 0 failures
20 carrier transitions
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
Serial1/0/1 -
Mx T3+(2) HW Revision 0x3, FW Revision 3.9
Framing is m13, Clock Source is Line
Bandwidth limit is 44210, DSU mode 0, Cable length is 100
Scrambling is enabled
Data in current interval (204 seconds elapsed):
0 Line Code Violations, 0 P-bit Coding Violation
0 C-bit Coding Violation
0 P-bit Err Secs, 0 P-bit Sev Err Secs
0 Sev Err Framing Secs, 34 Unavailable Secs
0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
Total Data (last 1 15 minute intervals):
0 Line Code Violations, 0 P-bit Coding Violation,
0 C-bit Coding Violation,
0 P-bit Err Secs, 0 P-bit Sev Err Secs,
0 Sev Err Framing Secs, 17 Unavailable Secs,
0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
No alarms detected.
Tabular MIB :
INTERVAL LCV PCV CCV PES PSES SEFS UAS LES CES CSES
05:21-05:25 0 0 0 0 0 0 34 0 0 0
05:06-05:21 0 0 0 0 0 0 17 0 0 0
R2's side:
Serial1/0 is up, line protocol is up
Hardware is M2T-T3+ pa
Internet address is 205.134.160.26/30
MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
reliability 255/255, txload 18/255, rxload 134/255
Encapsulation HDLC, crc 16, loopback not set
Keepalive set (10 sec)
Last input 00:00:01, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:15:13
Queueing strategy: fifo
Output queue 0/80, 0 drops; input queue 0/75, 0 drops
5 minute input rate 23301000 bits/sec, 4347 packets/sec
5 minute output rate 3269000 bits/sec, 4002 packets/sec
3938815 packets input, 2666550852 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
3629678 packets output, 365244110 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
M2T-T3+ pa: show controller:
PAS unit 0, subunit 0, f/w version 3-9, rev ID 0x2800001, version 3
idb = 0x6190B948, ds = 0x6190D3E8, ssb=0x6190D6B8
Clock mux=0x30, ucmd_ctrl=0x0, port_status=0x1
Serial config=0x8, line config=0x1B0202
maxdgram=4584, bufpool=128Kb, 256 particles
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
line state: up
cable type : T3 cable, received clockrate 44202795
base0 registers=0x3C800000, base1 registers=0x3C802000
mxt_ds=0x63600E3C, rx ring entries=124, tx ring entries=254
statring=0x2000AFC0, statr shadow=0x6190F4D8, stat_head=321
rxring=0x2000AD80, rxr shadow=0x6190EEAC, rx_head=24
txring=0x2000B600, txr shadow=0x61910104, tx_head=225, tx_tail=225,
tx_count=0
throttled=0, enabled=0
rx_no_eop_err=0, rx_no_stp_err=0, rx_no_eop_stp_err=0
rx_no_buf=0, rx_soft_overrun_err=2270, dump_err= 0, bogus=0, mxt_flags=0xC
tx_underrun_err=0, tx_soft_underrun_err=0, tx_limited=0(254)
tx_fullring=6677214, tx_started=-2027600724
Framing is m13, Clock Source is Line
Bandwidth limit is 44210, DSU mode 0, Cable length is 450
Scrambling is enabled
Data in current interval (92 seconds elapsed):
0 Line Code Violations, 0 P-bit Coding Violation
0 C-bit Coding Violation
0 P-bit Err Secs, 0 P-bit Sev Err Secs
0 Sev Err Framing Secs, 4 Unavailable Secs
0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
Total Data (last 1 15 minute intervals):
0 Line Code Violations, 0 P-bit Coding Violation,
0 C-bit Coding Violation,
0 P-bit Err Secs, 0 P-bit Sev Err Secs,
0 Sev Err Framing Secs, 9 Unavailable Secs,
0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
No alarms detected.
Serial1/1 is up, line protocol is up
Hardware is M2T-T3+ pa
Internet address is 205.134.160.22/30
MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
reliability 255/255, txload 17/255, rxload 147/255
Encapsulation HDLC, crc 16, loopback not set
Keepalive set (10 sec)
Last input 00:00:01, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:15:40
Queueing strategy: fifo
Output queue 0/80, 0 drops; input queue 0/75, 0 drops
5 minute input rate 25545000 bits/sec, 4658 packets/sec
5 minute output rate 2988000 bits/sec, 3973 packets/sec
4468724 packets input, 3074703657 bytes, 0 no buffer
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
3808888 packets output, 366478130 bytes, 0 underruns
0 output errors, 0 applique, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI inactive, txRAI inactive
M2T-T3+ pa: show controller:
PAS unit 1, subunit 1, f/w version 3-9, rev ID 0x2800001, version 3
idb = 0x61916784, ds = 0x61918224, ssb=0x619184F4
Clock mux=0x30, ucmd_ctrl=0x0, port_status=0x1
Serial config=0x8, line config=0x1B0202
maxdgram=4584, bufpool=128Kb, 256 particles
rxLOS inactive, rxLOF inactive, rxAIS inactive
txAIS inactive, rxRAI active, txRAI inactive
line state: up
cable type : T3 cable, received clockrate 44201430
base0 registers=0x3C804000, base1 registers=0x3C806000
mxt_ds=0x63600E3C, rx ring entries=124, tx ring entries=254
statring=0x2000C2C0, statr shadow=0x6191A2C0, stat_head=249
rxring=0x2000C080, rxr shadow=0x61919C94, rx_head=28
txring=0x2000C900, txr shadow=0x6191AEEC, tx_head=70, tx_tail=71, tx_count=1
throttled=0, enabled=0
rx_no_eop_err=0, rx_no_stp_err=0, rx_no_eop_stp_err=0
rx_no_buf=0, rx_soft_overrun_err=39493, dump_err= 0, bogus=0, mxt_flags=0xC
tx_underrun_err=0, tx_soft_underrun_err=0, tx_limited=0(254)
tx_fullring=1264480, tx_started=-1999548355
Framing is m13, Clock Source is Line
Bandwidth limit is 44210, DSU mode 0, Cable length is 450
Scrambling is enabled
Data in current interval (132 seconds elapsed):
0 Line Code Violations, 0 P-bit Coding Violation
0 C-bit Coding Violation
0 P-bit Err Secs, 0 P-bit Sev Err Secs
0 Sev Err Framing Secs, 6 Unavailable Secs
0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
Total Data (last 1 15 minute intervals):
0 Line Code Violations, 0 P-bit Coding Violation,
0 C-bit Coding Violation,
0 P-bit Err Secs, 0 P-bit Sev Err Secs,
0 Sev Err Framing Secs, 9 Unavailable Secs,
0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
No alarms detected.
This archive was generated by hypermail 2b29 : Sun Aug 04 2002 - 04:12:39 EDT