[c-nsp] RSP-6-TXSTUCK - software or hardware?
David Freedman
david.freedman at uk.clara.net
Fri Aug 11 09:57:30 EDT 2006
Ok, interface is up, and is definately CEF switching:
Serial6/0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 1 64 1 273
Route cache 0 0 0 0
Distributed cache 12672 5424852 0 0
Total 12673 5424916 1 273
Thanks,
Dave.
Rodney Dunn wrote:
> Wait until it's back up. Clear the counters and let it run
> for a bit and send me the output again.
>
>
> If it's not showing as distributed or route-cache in 'sh int stat'
> we have to look at the features you have configured to see why
> it's punting.
>
> On Fri, Aug 11, 2006 at 02:25:12PM +0100, David Freedman wrote:
>> oh, just in case you are wondering, the interface is currently down
>> (transmission people are having another go at the cabling)
>> but in case you are concerned by the "stat" output (does not show CEF
>> for some reason) it should definately be CEF switching:
>>
>> #sh cef int s6/0/0
>> Serial6/0/0 is down (if_number 6)
>> Corresponding hwidb fast_if_number 6
>> Corresponding hwidb firstsw->if_number 6
>> Internet address is x.x.x.x/y
>> ICMP redirects are always sent
>> Per packet load-sharing is disabled
>> IP unicast RPF check is disabled
>> Inbound access list is not set
>> Outbound access list is not set
>> IP policy routing is disabled
>> BGP based policy accounting is disabled
>> Interface is marked as point to point interface
>> Hardware idb is Serial6/0/0
>> Fast switching type 4, interface type 11
>> IP Distributed CEF switching enabled
>> IP Flow switching turbo vector
>> IP Flow CEF switching turbo vector
>> Input fast flags 0x0, Output fast flags 0x0
>> ifindex 5(5)
>> Slot 6 Slot unit 0 Unit 0 VC -1
>> Transmit limit accumulator 0xE8001A82 (0xE8001A82)
>> IP MTU 4470
>>
>> #execute-on slot 6 show cef int stat | in Serial6
>> show cef int stat | in Serial6 from slot 6:
>> Serial6/0/0 2557769059 540490892955 4558247 933422662
>>
>> Thanks,
>>
>> Dave.
>>
>>
>> David Freedman wrote:
>> > Sure:
>> >
>> > Rodney Dunn wrote:
>> >> David,
>> >>
>> >> What those messages were designed for was to help
>> >> identify if there was tx accumulator leaks on a 75xx before
>> >> it started affecting traffic.
>> >> It was one of the code areas we had some bugs so we put
>> >> that in to try and help. However it caused some false positives
>> >> so we removed it for E1's and below from what I remember.
>> >>
>> >> Pretty unilikely your load is high enough to get to the DS3 rate
>> >> as long as it's really clocking at a T3.
>> >>
>> >> Can you provide:
>> >> sh int se 6/0/0 stat
>> >
>> > #sh int s6/0/0 stat
>> > Serial6/0/0
>> > Switching path Pkts In Chars In Pkts Out Chars Out
>> > Processor 20039 2084968 20039 2084968
>> > Route cache 0 0 0 0
>> > Distributed cache 0 0 0 0
>> > Total 20039 2084968 20039 2084968
>> >
>> >
>> >
>> >
>> >
>> >>
>> >> and sh contr cbus
>> >
>> >
>> > #show controllers cbus | beg slot6
>> > slot6: VIP2 R5K, hw 2.00, sw 22.20, ccb F800FF70, cmdq E80000B0, vps 8192
>> > software loaded from system
>> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10d), RELEASE
>> > SOFTWARE (fc2)
>> > ROM Monitor version 115.0
>> > Mx T3+(1) HW Revision 0x3, FW Revision 3.93
>> > Serial6/0/0, applique is T3+ DTE
>> > received clockrate 1365
>> > gfreeq E8000180, lfreeq E80001B0 (4512 bytes)
>> > rxlo 4, rxhi 78, rxcurr 0, maxrxcurr 3
>> > txq E8001A80, txacc E8001A82 (value 76), txlimit 78
>> > FastEthernet6/1/0, addr 0001.9700.78c8 (bia 0001.9700.78c8)
>> > gfreeq E8000160, lfreeq E80001B8 (1536 bytes)
>> > rxlo 4, rxhi 526, rxcurr 4, maxrxcurr 29
>> > txq E8001A88, txacc E8001A8A (value 264), txlimit 265
>> >
>> >
>> >>
>> >> from the box.
>> >>
>> >> See what the value/limit is for that serial 6/0/0 interface.
>> >>
>> >> There is a remote remote remote chance it could be bad hadware
>> >> causing buffer header corruption which results in txacc leaks.
>> >> I've only seen that 3 times every in 8 years but it is a chance.
>> >> Almost always it's a software bug.
>> >>
>> >> Rodney
>> >>
>> >> On Thu, Aug 10, 2006 at 01:15:06PM +0100, David Freedman wrote:
>> >>>
>> >>> Hardware: 75xx(dual RSP4) with VIP2-50 populated with PA-T3+ and PA-FE-TX
>> >>>
>> >>> What would cause the following messages to occur?
>> >>>
>> >>>
>> >>> Aug 9 21:13:29 BST: %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is
>> >>> at 2% of its txlimit
>> >>>
>> >>> Aug 10 10:08:48 BST: %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is
>> >>> at 3% of its txlimit
>> >>>
>> >>> Aug 10 12:02:00 BST: %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is
>> >>> at 2% of its txlimit
>> >>>
>> >>>
>> >>> Traffic on this interface is miniscule (i.e around 8/10Mbps) and on the
>> >>> FE interface around 16/20Mbps, both with appropriate packet rates.
>> >>>
>> >>> Checking the graphs for the times that these events were reported ,
>> >>> nothing untoward was occuring at the time.
>> >>>
>> >>> We also graph the CPU of the VIPs and the VIP didn't rise about 10% CPU
>> >>> at these times.
>> >>>
>> >>> The box is a legacy piece of kit which has been beefed up somewhat to
>> >>> run 12.3 (as it has IPv6 requirements) , its currently running 12.3(10)d
>> >>> but I can't find any known bugs that refer directly to this.
>> >>>
>> >>> The T3 has been experiencing Input errors for around a week before these
>> >>> messages were appearing:
>> >>>
>> >>> Serial6/0/0 is up, line protocol is up
>> >>> Hardware is cyBus PODS3+ Serial
>> >>> Description: Protection T path DS3 over 100239/450845-0/2308
>> >>> MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
>> >>> reliability 255/255, txload 5/255, rxload 39/255
>> >>> Encapsulation HDLC, crc 16, loopback not set
>> >>> Keepalive set (10 sec)
>> >>> Restart-Delay is 0 secs
>> >>> Last input 00:00:00, output 00:00:00, output hang never
>> >>> Last clearing of "show interface" counters 1d19h
>> >>> Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops:
>> >>> 9714
>> >>> Queueing strategy: fifo
>> >>> Output queue: 0/40 (size/max)
>> >>> 30 second input rate 6825000 bits/sec, 1325 packets/sec
>> >>> 30 second output rate 914000 bits/sec, 320 packets/sec
>> >>> 610766319 packets input, 3114706794 bytes, 0 no buffer
>> >>> Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
>> >>> 0 parity
>> >>> 158422 input errors, 137421 CRC, 0 frame, 18741 overrun, 16
>> >>> ignored, 2260 abort
>> >>> 34167551 packets output, 1658107765 bytes, 0 underruns
>> >>> 0 output errors, 0 applique, 0 interface resets
>> >>> 0 output buffer failures, 0 output buffers swapped out
>> >>> 236 carrier transitions
>> >>> rxLOS inactive, rxLOF inactive, rxAIS inactive
>> >>> txAIS inactive, rxRAI inactive, txRAI inactive
>> >>>
>> >>>
>> >>> I've been blaming the transmission people for the past week , but they
>> >>> are as mystified as me, is it possible the TXSTUCK messages are a result
>> >>> of some kind of events occuring on the transmission network?
>> >>>
>> >>> For reference, its configured as a point-to-point DS3, as so:
>> >>>
>> >>> interface Serial6/0/0
>> >>> description Protection T path DS3 over 100239/450845-0/2308
>> >>> load-interval 30
>> >>> dsu bandwidth 44210
>> >>> scramble
>> >>> framing c-bit
>> >>> cablelength 50
>> >>> serial restart-delay 0
>> >>> no clns route-cache
>> >>> hold-queue 2000 in
>> >>> end
>> >>>
>> >>>
>> >>>
>> >>> If we look at the PA-FE:
>> >>>
>> >>>
>> >>> FastEthernet6/1/0 is up, line protocol is up
>> >>> Hardware is cyBus FastEthernet Interface, address is 0001.9700.78c8
>> >>> (bia 0001.9700.78c8)
>> >>> Description: Worker ET Path 100239/450845-0/2307
>> >>> MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
>> >>> reliability 255/255, txload 39/255, rxload 13/255
>> >>> Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
>> >>> Keepalive set (10 sec)
>> >>> Full-duplex, 100Mb/s, 100BaseTX/FX
>> >>> ARP type: ARPA, ARP Timeout 04:00:00
>> >>> Last input 00:00:00, output 00:00:06, output hang never
>> >>> Last clearing of "show interface" counters never
>> >>> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
>> >>> Queueing strategy: fifo
>> >>> Output queue: 0/40 (size/max)
>> >>> 5 minute input rate 5381000 bits/sec, 2258 packets/sec
>> >>> 5 minute output rate 15681000 bits/sec, 2623 packets/sec
>> >>> 1271168625 packets input, 2747320900 bytes, 0 no buffer
>> >>> Received 14588 broadcasts, 0 runts, 0 giants, 0 throttles
>> >>> 36 input errors, 0 CRC, 0 frame, 0 overrun, 36 ignored
>> >>> 0 watchdog
>> >>> 0 input packets with dribble condition detected
>> >>> 1476558580 packets output, 3223841169 bytes, 0 underruns
>> >>> 0 output errors, 0 collisions, 0 interface resets
>> >>> 0 babbles, 0 late collision, 0 deferred
>> >>> 0 lost carrier, 0 no carrier
>> >>> 0 output buffer failures, 0 output buffers swapped out
>> >>>
>> >>>
>> >>> Nothing out of the ordinary.
>> >>>
>> >>>
>> >>> Not quite sure how to proceed, I'm considering replacing the PA and/or
>> >>> the VIP but if anybody can suggest this as being a s/w oddity I'm
>> >>> willing to replace/upgrade as needed (bugtool is not turning up anything
>> >>> of interest here)
>> >>>
>> >>>
>> >>> Any help/advice on this muchly appreciated.
>> >>>
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Dave.
>> >>>
>> >>> _______________________________________________
>> >>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> >>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> >>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>> >> _______________________________________________
>> >> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> >> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> >> archive at http://puck.nether.net/pipermail/cisco-nsp/
>> >>
>> >
>> > _______________________________________________
>> > cisco-nsp mailing list cisco-nsp at puck.nether.net
>> > https://puck.nether.net/mailman/listinfo/cisco-nsp
>> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>> >
>>
>> _______________________________________________
>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
More information about the cisco-nsp
mailing list