[c-nsp] RSP-6-TXSTUCK - software or hardware?
David Freedman
david.freedman at uk.clara.net
Fri Aug 11 10:41:19 EDT 2006
Rodney Dunn wrote:
> Where are the out packets?
Sorry, I was a bit quick to get the output after the interface had come
back, they are definately there :)
>
> Now watch for the message and monitor the sh contr cbus
> to see if it goes down.
Shall do
I've noticed that its started happening to other cards as well:
%RSP-6-TXSTUCK: Txacc of Interface FastEthernet5/1/0 is at 1% of its txlimit
%RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is at 2% of its txlimit
%RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is at 3% of its txlimit
%RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is at 2% of its txlimit
%RSP-6-TXSTUCK: Txacc of Interface FastEthernet6/1/0 is at 0% of its txlimit
I shall keep an eye on that counter in the "sh contr cbus" next time
this happens, but it will be hard to catch
thanks again.
Dave.
>
> On Fri, Aug 11, 2006 at 02:57:30PM +0100, David Freedman wrote:
>> Ok, interface is up, and is definately CEF switching:
>>
>> Serial6/0/0
>> Switching path Pkts In Chars In Pkts Out Chars Out
>> Processor 1 64 1 273
>> Route cache 0 0 0 0
>> Distributed cache 12672 5424852 0 0
>> Total 12673 5424916 1 273
>>
>>
>> Thanks,
>>
>> Dave.
>> Rodney Dunn wrote:
>> > Wait until it's back up. Clear the counters and let it run
>> > for a bit and send me the output again.
>> >
>> >
>> > If it's not showing as distributed or route-cache in 'sh int stat'
>> > we have to look at the features you have configured to see why
>> > it's punting.
>> >
>> > On Fri, Aug 11, 2006 at 02:25:12PM +0100, David Freedman wrote:
>> >> oh, just in case you are wondering, the interface is currently down
>> >> (transmission people are having another go at the cabling)
>> >> but in case you are concerned by the "stat" output (does not show CEF
>> >> for some reason) it should definately be CEF switching:
>> >>
>> >> #sh cef int s6/0/0
>> >> Serial6/0/0 is down (if_number 6)
>> >> Corresponding hwidb fast_if_number 6
>> >> Corresponding hwidb firstsw->if_number 6
>> >> Internet address is x.x.x.x/y
>> >> ICMP redirects are always sent
>> >> Per packet load-sharing is disabled
>> >> IP unicast RPF check is disabled
>> >> Inbound access list is not set
>> >> Outbound access list is not set
>> >> IP policy routing is disabled
>> >> BGP based policy accounting is disabled
>> >> Interface is marked as point to point interface
>> >> Hardware idb is Serial6/0/0
>> >> Fast switching type 4, interface type 11
>> >> IP Distributed CEF switching enabled
>> >> IP Flow switching turbo vector
>> >> IP Flow CEF switching turbo vector
>> >> Input fast flags 0x0, Output fast flags 0x0
>> >> ifindex 5(5)
>> >> Slot 6 Slot unit 0 Unit 0 VC -1
>> >> Transmit limit accumulator 0xE8001A82 (0xE8001A82)
>> >> IP MTU 4470
>> >>
>> >> #execute-on slot 6 show cef int stat | in Serial6
>> >> show cef int stat | in Serial6 from slot 6:
>> >> Serial6/0/0 2557769059 540490892955 4558247 933422662
>> >>
>> >> Thanks,
>> >>
>> >> Dave.
>> >>
>> >>
>> >> David Freedman wrote:
>> >> > Sure:
>> >> >
>> >> > Rodney Dunn wrote:
>> >> >> David,
>> >> >>
>> >> >> What those messages were designed for was to help
>> >> >> identify if there was tx accumulator leaks on a 75xx before
>> >> >> it started affecting traffic.
>> >> >> It was one of the code areas we had some bugs so we put
>> >> >> that in to try and help. However it caused some false positives
>> >> >> so we removed it for E1's and below from what I remember.
>> >> >>
>> >> >> Pretty unilikely your load is high enough to get to the DS3 rate
>> >> >> as long as it's really clocking at a T3.
>> >> >>
>> >> >> Can you provide:
>> >> >> sh int se 6/0/0 stat
>> >> >
>> >> > #sh int s6/0/0 stat
>> >> > Serial6/0/0
>> >> > Switching path Pkts In Chars In Pkts Out Chars Out
>> >> > Processor 20039 2084968 20039 2084968
>> >> > Route cache 0 0 0 0
>> >> > Distributed cache 0 0 0 0
>> >> > Total 20039 2084968 20039 2084968
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >>
>> >> >> and sh contr cbus
>> >> >
>> >> >
>> >> > #show controllers cbus | beg slot6
>> >> > slot6: VIP2 R5K, hw 2.00, sw 22.20, ccb F800FF70, cmdq E80000B0, vps 8192
>> >> > software loaded from system
>> >> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10d), RELEASE
>> >> > SOFTWARE (fc2)
>> >> > ROM Monitor version 115.0
>> >> > Mx T3+(1) HW Revision 0x3, FW Revision 3.93
>> >> > Serial6/0/0, applique is T3+ DTE
>> >> > received clockrate 1365
>> >> > gfreeq E8000180, lfreeq E80001B0 (4512 bytes)
>> >> > rxlo 4, rxhi 78, rxcurr 0, maxrxcurr 3
>> >> > txq E8001A80, txacc E8001A82 (value 76), txlimit 78
>> >> > FastEthernet6/1/0, addr 0001.9700.78c8 (bia 0001.9700.78c8)
>> >> > gfreeq E8000160, lfreeq E80001B8 (1536 bytes)
>> >> > rxlo 4, rxhi 526, rxcurr 4, maxrxcurr 29
>> >> > txq E8001A88, txacc E8001A8A (value 264), txlimit 265
>> >> >
>> >> >
>> >> >>
>> >> >> from the box.
>> >> >>
>> >> >> See what the value/limit is for that serial 6/0/0 interface.
>> >> >>
>> >> >> There is a remote remote remote chance it could be bad hadware
>> >> >> causing buffer header corruption which results in txacc leaks.
>> >> >> I've only seen that 3 times every in 8 years but it is a chance.
>> >> >> Almost always it's a software bug.
>> >> >>
>> >> >> Rodney
>> >> >>
>> >> >> On Thu, Aug 10, 2006 at 01:15:06PM +0100, David Freedman wrote:
>> >> >>>
>> >> >>> Hardware: 75xx(dual RSP4) with VIP2-50 populated with PA-T3+ and PA-FE-TX
>> >> >>>
>> >> >>> What would cause the following messages to occur?
>> >> >>>
>> >> >>>
>> >> >>> Aug 9 21:13:29 BST: %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is
>> >> >>> at 2% of its txlimit
>> >> >>>
>> >> >>> Aug 10 10:08:48 BST: %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is
>> >> >>> at 3% of its txlimit
>> >> >>>
>> >> >>> Aug 10 12:02:00 BST: %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is
>> >> >>> at 2% of its txlimit
>> >> >>>
>> >> >>>
>> >> >>> Traffic on this interface is miniscule (i.e around 8/10Mbps) and on the
>> >> >>> FE interface around 16/20Mbps, both with appropriate packet rates.
>> >> >>>
>> >> >>> Checking the graphs for the times that these events were reported ,
>> >> >>> nothing untoward was occuring at the time.
>> >> >>>
>> >> >>> We also graph the CPU of the VIPs and the VIP didn't rise about 10% CPU
>> >> >>> at these times.
>> >> >>>
>> >> >>> The box is a legacy piece of kit which has been beefed up somewhat to
>> >> >>> run 12.3 (as it has IPv6 requirements) , its currently running 12.3(10)d
>> >> >>> but I can't find any known bugs that refer directly to this.
>> >> >>>
>> >> >>> The T3 has been experiencing Input errors for around a week before these
>> >> >>> messages were appearing:
>> >> >>>
>> >> >>> Serial6/0/0 is up, line protocol is up
>> >> >>> Hardware is cyBus PODS3+ Serial
>> >> >>> Description: Protection T path DS3 over 100239/450845-0/2308
>> >> >>> MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
>> >> >>> reliability 255/255, txload 5/255, rxload 39/255
>> >> >>> Encapsulation HDLC, crc 16, loopback not set
>> >> >>> Keepalive set (10 sec)
>> >> >>> Restart-Delay is 0 secs
>> >> >>> Last input 00:00:00, output 00:00:00, output hang never
>> >> >>> Last clearing of "show interface" counters 1d19h
>> >> >>> Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops:
>> >> >>> 9714
>> >> >>> Queueing strategy: fifo
>> >> >>> Output queue: 0/40 (size/max)
>> >> >>> 30 second input rate 6825000 bits/sec, 1325 packets/sec
>> >> >>> 30 second output rate 914000 bits/sec, 320 packets/sec
>> >> >>> 610766319 packets input, 3114706794 bytes, 0 no buffer
>> >> >>> Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
>> >> >>> 0 parity
>> >> >>> 158422 input errors, 137421 CRC, 0 frame, 18741 overrun, 16
>> >> >>> ignored, 2260 abort
>> >> >>> 34167551 packets output, 1658107765 bytes, 0 underruns
>> >> >>> 0 output errors, 0 applique, 0 interface resets
>> >> >>> 0 output buffer failures, 0 output buffers swapped out
>> >> >>> 236 carrier transitions
>> >> >>> rxLOS inactive, rxLOF inactive, rxAIS inactive
>> >> >>> txAIS inactive, rxRAI inactive, txRAI inactive
>> >> >>>
>> >> >>>
>> >> >>> I've been blaming the transmission people for the past week , but they
>> >> >>> are as mystified as me, is it possible the TXSTUCK messages are a result
>> >> >>> of some kind of events occuring on the transmission network?
>> >> >>>
>> >> >>> For reference, its configured as a point-to-point DS3, as so:
>> >> >>>
>> >> >>> interface Serial6/0/0
>> >> >>> description Protection T path DS3 over 100239/450845-0/2308
>> >> >>> load-interval 30
>> >> >>> dsu bandwidth 44210
>> >> >>> scramble
>> >> >>> framing c-bit
>> >> >>> cablelength 50
>> >> >>> serial restart-delay 0
>> >> >>> no clns route-cache
>> >> >>> hold-queue 2000 in
>> >> >>> end
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> If we look at the PA-FE:
>> >> >>>
>> >> >>>
>> >> >>> FastEthernet6/1/0 is up, line protocol is up
>> >> >>> Hardware is cyBus FastEthernet Interface, address is 0001.9700.78c8
>> >> >>> (bia 0001.9700.78c8)
>> >> >>> Description: Worker ET Path 100239/450845-0/2307
>> >> >>> MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
>> >> >>> reliability 255/255, txload 39/255, rxload 13/255
>> >> >>> Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
>> >> >>> Keepalive set (10 sec)
>> >> >>> Full-duplex, 100Mb/s, 100BaseTX/FX
>> >> >>> ARP type: ARPA, ARP Timeout 04:00:00
>> >> >>> Last input 00:00:00, output 00:00:06, output hang never
>> >> >>> Last clearing of "show interface" counters never
>> >> >>> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
>> >> >>> Queueing strategy: fifo
>> >> >>> Output queue: 0/40 (size/max)
>> >> >>> 5 minute input rate 5381000 bits/sec, 2258 packets/sec
>> >> >>> 5 minute output rate 15681000 bits/sec, 2623 packets/sec
>> >> >>> 1271168625 packets input, 2747320900 bytes, 0 no buffer
>> >> >>> Received 14588 broadcasts, 0 runts, 0 giants, 0 throttles
>> >> >>> 36 input errors, 0 CRC, 0 frame, 0 overrun, 36 ignored
>> >> >>> 0 watchdog
>> >> >>> 0 input packets with dribble condition detected
>> >> >>> 1476558580 packets output, 3223841169 bytes, 0 underruns
>> >> >>> 0 output errors, 0 collisions, 0 interface resets
>> >> >>> 0 babbles, 0 late collision, 0 deferred
>> >> >>> 0 lost carrier, 0 no carrier
>> >> >>> 0 output buffer failures, 0 output buffers swapped out
>> >> >>>
>> >> >>>
>> >> >>> Nothing out of the ordinary.
>> >> >>>
>> >> >>>
>> >> >>> Not quite sure how to proceed, I'm considering replacing the PA and/or
>> >> >>> the VIP but if anybody can suggest this as being a s/w oddity I'm
>> >> >>> willing to replace/upgrade as needed (bugtool is not turning up anything
>> >> >>> of interest here)
>> >> >>>
>> >> >>>
>> >> >>> Any help/advice on this muchly appreciated.
>> >> >>>
>> >> >>>
>> >> >>> Thanks,
>> >> >>>
>> >> >>> Dave.
>> >> >>>
>> >> >>> _______________________________________________
>> >> >>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> >> >>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> >> >>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>> >> >> _______________________________________________
>> >> >> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> >> >> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> >> >> archive at http://puck.nether.net/pipermail/cisco-nsp/
>> >> >>
>> >> >
>> >> > _______________________________________________
>> >> > cisco-nsp mailing list cisco-nsp at puck.nether.net
>> >> > https://puck.nether.net/mailman/listinfo/cisco-nsp
>> >> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>> >> >
>> >>
>> >> _______________________________________________
>> >> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> >> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> >> archive at http://puck.nether.net/pipermail/cisco-nsp/
>> > _______________________________________________
>> > cisco-nsp mailing list cisco-nsp at puck.nether.net
>> > https://puck.nether.net/mailman/listinfo/cisco-nsp
>> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>> >
>>
>> _______________________________________________
>> cisco-nsp mailing list cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
More information about the cisco-nsp
mailing list