[c-nsp] RSP-6-TXSTUCK - software or hardware?
Rodney Dunn
rodunn at cisco.com
Fri Aug 11 15:09:09 EDT 2006
If you have real txacc loss it will NOT recover on it's own.
so if that's the problem you'll know it.
On Fri, Aug 11, 2006 at 03:41:19PM +0100, David Freedman wrote:
>
> Rodney Dunn wrote:
> > Where are the out packets?
>
> Sorry, I was a bit quick to get the output after the interface had come
> back, they are definately there :)
>
>
> >
> > Now watch for the message and monitor the sh contr cbus
> > to see if it goes down.
>
> Shall do
>
> I've noticed that its started happening to other cards as well:
>
>
> %RSP-6-TXSTUCK: Txacc of Interface FastEthernet5/1/0 is at 1% of its txlimit
>
> %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is at 2% of its txlimit
>
> %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is at 3% of its txlimit
>
> %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is at 2% of its txlimit
>
> %RSP-6-TXSTUCK: Txacc of Interface FastEthernet6/1/0 is at 0% of its txlimit
>
> I shall keep an eye on that counter in the "sh contr cbus" next time
> this happens, but it will be hard to catch
>
> thanks again.
>
> Dave.
>
> >
> > On Fri, Aug 11, 2006 at 02:57:30PM +0100, David Freedman wrote:
> >> Ok, interface is up, and is definately CEF switching:
> >>
> >> Serial6/0/0
> >> Switching path Pkts In Chars In Pkts Out Chars Out
> >> Processor 1 64 1 273
> >> Route cache 0 0 0 0
> >> Distributed cache 12672 5424852 0 0
> >> Total 12673 5424916 1 273
> >>
> >>
> >> Thanks,
> >>
> >> Dave.
> >> Rodney Dunn wrote:
> >> > Wait until it's back up. Clear the counters and let it run
> >> > for a bit and send me the output again.
> >> >
> >> >
> >> > If it's not showing as distributed or route-cache in 'sh int stat'
> >> > we have to look at the features you have configured to see why
> >> > it's punting.
> >> >
> >> > On Fri, Aug 11, 2006 at 02:25:12PM +0100, David Freedman wrote:
> >> >> oh, just in case you are wondering, the interface is currently down
> >> >> (transmission people are having another go at the cabling)
> >> >> but in case you are concerned by the "stat" output (does not show CEF
> >> >> for some reason) it should definately be CEF switching:
> >> >>
> >> >> #sh cef int s6/0/0
> >> >> Serial6/0/0 is down (if_number 6)
> >> >> Corresponding hwidb fast_if_number 6
> >> >> Corresponding hwidb firstsw->if_number 6
> >> >> Internet address is x.x.x.x/y
> >> >> ICMP redirects are always sent
> >> >> Per packet load-sharing is disabled
> >> >> IP unicast RPF check is disabled
> >> >> Inbound access list is not set
> >> >> Outbound access list is not set
> >> >> IP policy routing is disabled
> >> >> BGP based policy accounting is disabled
> >> >> Interface is marked as point to point interface
> >> >> Hardware idb is Serial6/0/0
> >> >> Fast switching type 4, interface type 11
> >> >> IP Distributed CEF switching enabled
> >> >> IP Flow switching turbo vector
> >> >> IP Flow CEF switching turbo vector
> >> >> Input fast flags 0x0, Output fast flags 0x0
> >> >> ifindex 5(5)
> >> >> Slot 6 Slot unit 0 Unit 0 VC -1
> >> >> Transmit limit accumulator 0xE8001A82 (0xE8001A82)
> >> >> IP MTU 4470
> >> >>
> >> >> #execute-on slot 6 show cef int stat | in Serial6
> >> >> show cef int stat | in Serial6 from slot 6:
> >> >> Serial6/0/0 2557769059 540490892955 4558247 933422662
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Dave.
> >> >>
> >> >>
> >> >> David Freedman wrote:
> >> >> > Sure:
> >> >> >
> >> >> > Rodney Dunn wrote:
> >> >> >> David,
> >> >> >>
> >> >> >> What those messages were designed for was to help
> >> >> >> identify if there was tx accumulator leaks on a 75xx before
> >> >> >> it started affecting traffic.
> >> >> >> It was one of the code areas we had some bugs so we put
> >> >> >> that in to try and help. However it caused some false positives
> >> >> >> so we removed it for E1's and below from what I remember.
> >> >> >>
> >> >> >> Pretty unilikely your load is high enough to get to the DS3 rate
> >> >> >> as long as it's really clocking at a T3.
> >> >> >>
> >> >> >> Can you provide:
> >> >> >> sh int se 6/0/0 stat
> >> >> >
> >> >> > #sh int s6/0/0 stat
> >> >> > Serial6/0/0
> >> >> > Switching path Pkts In Chars In Pkts Out Chars Out
> >> >> > Processor 20039 2084968 20039 2084968
> >> >> > Route cache 0 0 0 0
> >> >> > Distributed cache 0 0 0 0
> >> >> > Total 20039 2084968 20039 2084968
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >>
> >> >> >> and sh contr cbus
> >> >> >
> >> >> >
> >> >> > #show controllers cbus | beg slot6
> >> >> > slot6: VIP2 R5K, hw 2.00, sw 22.20, ccb F800FF70, cmdq E80000B0, vps 8192
> >> >> > software loaded from system
> >> >> > IOS (tm) VIP Software (SVIP-DW-M), Version 12.3(10d), RELEASE
> >> >> > SOFTWARE (fc2)
> >> >> > ROM Monitor version 115.0
> >> >> > Mx T3+(1) HW Revision 0x3, FW Revision 3.93
> >> >> > Serial6/0/0, applique is T3+ DTE
> >> >> > received clockrate 1365
> >> >> > gfreeq E8000180, lfreeq E80001B0 (4512 bytes)
> >> >> > rxlo 4, rxhi 78, rxcurr 0, maxrxcurr 3
> >> >> > txq E8001A80, txacc E8001A82 (value 76), txlimit 78
> >> >> > FastEthernet6/1/0, addr 0001.9700.78c8 (bia 0001.9700.78c8)
> >> >> > gfreeq E8000160, lfreeq E80001B8 (1536 bytes)
> >> >> > rxlo 4, rxhi 526, rxcurr 4, maxrxcurr 29
> >> >> > txq E8001A88, txacc E8001A8A (value 264), txlimit 265
> >> >> >
> >> >> >
> >> >> >>
> >> >> >> from the box.
> >> >> >>
> >> >> >> See what the value/limit is for that serial 6/0/0 interface.
> >> >> >>
> >> >> >> There is a remote remote remote chance it could be bad hadware
> >> >> >> causing buffer header corruption which results in txacc leaks.
> >> >> >> I've only seen that 3 times every in 8 years but it is a chance.
> >> >> >> Almost always it's a software bug.
> >> >> >>
> >> >> >> Rodney
> >> >> >>
> >> >> >> On Thu, Aug 10, 2006 at 01:15:06PM +0100, David Freedman wrote:
> >> >> >>>
> >> >> >>> Hardware: 75xx(dual RSP4) with VIP2-50 populated with PA-T3+ and PA-FE-TX
> >> >> >>>
> >> >> >>> What would cause the following messages to occur?
> >> >> >>>
> >> >> >>>
> >> >> >>> Aug 9 21:13:29 BST: %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is
> >> >> >>> at 2% of its txlimit
> >> >> >>>
> >> >> >>> Aug 10 10:08:48 BST: %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is
> >> >> >>> at 3% of its txlimit
> >> >> >>>
> >> >> >>> Aug 10 12:02:00 BST: %RSP-6-TXSTUCK: Txacc of Interface Serial6/0/0 is
> >> >> >>> at 2% of its txlimit
> >> >> >>>
> >> >> >>>
> >> >> >>> Traffic on this interface is miniscule (i.e around 8/10Mbps) and on the
> >> >> >>> FE interface around 16/20Mbps, both with appropriate packet rates.
> >> >> >>>
> >> >> >>> Checking the graphs for the times that these events were reported ,
> >> >> >>> nothing untoward was occuring at the time.
> >> >> >>>
> >> >> >>> We also graph the CPU of the VIPs and the VIP didn't rise about 10% CPU
> >> >> >>> at these times.
> >> >> >>>
> >> >> >>> The box is a legacy piece of kit which has been beefed up somewhat to
> >> >> >>> run 12.3 (as it has IPv6 requirements) , its currently running 12.3(10)d
> >> >> >>> but I can't find any known bugs that refer directly to this.
> >> >> >>>
> >> >> >>> The T3 has been experiencing Input errors for around a week before these
> >> >> >>> messages were appearing:
> >> >> >>>
> >> >> >>> Serial6/0/0 is up, line protocol is up
> >> >> >>> Hardware is cyBus PODS3+ Serial
> >> >> >>> Description: Protection T path DS3 over 100239/450845-0/2308
> >> >> >>> MTU 4470 bytes, BW 44210 Kbit, DLY 200 usec,
> >> >> >>> reliability 255/255, txload 5/255, rxload 39/255
> >> >> >>> Encapsulation HDLC, crc 16, loopback not set
> >> >> >>> Keepalive set (10 sec)
> >> >> >>> Restart-Delay is 0 secs
> >> >> >>> Last input 00:00:00, output 00:00:00, output hang never
> >> >> >>> Last clearing of "show interface" counters 1d19h
> >> >> >>> Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops:
> >> >> >>> 9714
> >> >> >>> Queueing strategy: fifo
> >> >> >>> Output queue: 0/40 (size/max)
> >> >> >>> 30 second input rate 6825000 bits/sec, 1325 packets/sec
> >> >> >>> 30 second output rate 914000 bits/sec, 320 packets/sec
> >> >> >>> 610766319 packets input, 3114706794 bytes, 0 no buffer
> >> >> >>> Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
> >> >> >>> 0 parity
> >> >> >>> 158422 input errors, 137421 CRC, 0 frame, 18741 overrun, 16
> >> >> >>> ignored, 2260 abort
> >> >> >>> 34167551 packets output, 1658107765 bytes, 0 underruns
> >> >> >>> 0 output errors, 0 applique, 0 interface resets
> >> >> >>> 0 output buffer failures, 0 output buffers swapped out
> >> >> >>> 236 carrier transitions
> >> >> >>> rxLOS inactive, rxLOF inactive, rxAIS inactive
> >> >> >>> txAIS inactive, rxRAI inactive, txRAI inactive
> >> >> >>>
> >> >> >>>
> >> >> >>> I've been blaming the transmission people for the past week , but they
> >> >> >>> are as mystified as me, is it possible the TXSTUCK messages are a result
> >> >> >>> of some kind of events occuring on the transmission network?
> >> >> >>>
> >> >> >>> For reference, its configured as a point-to-point DS3, as so:
> >> >> >>>
> >> >> >>> interface Serial6/0/0
> >> >> >>> description Protection T path DS3 over 100239/450845-0/2308
> >> >> >>> load-interval 30
> >> >> >>> dsu bandwidth 44210
> >> >> >>> scramble
> >> >> >>> framing c-bit
> >> >> >>> cablelength 50
> >> >> >>> serial restart-delay 0
> >> >> >>> no clns route-cache
> >> >> >>> hold-queue 2000 in
> >> >> >>> end
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>> If we look at the PA-FE:
> >> >> >>>
> >> >> >>>
> >> >> >>> FastEthernet6/1/0 is up, line protocol is up
> >> >> >>> Hardware is cyBus FastEthernet Interface, address is 0001.9700.78c8
> >> >> >>> (bia 0001.9700.78c8)
> >> >> >>> Description: Worker ET Path 100239/450845-0/2307
> >> >> >>> MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
> >> >> >>> reliability 255/255, txload 39/255, rxload 13/255
> >> >> >>> Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
> >> >> >>> Keepalive set (10 sec)
> >> >> >>> Full-duplex, 100Mb/s, 100BaseTX/FX
> >> >> >>> ARP type: ARPA, ARP Timeout 04:00:00
> >> >> >>> Last input 00:00:00, output 00:00:06, output hang never
> >> >> >>> Last clearing of "show interface" counters never
> >> >> >>> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
> >> >> >>> Queueing strategy: fifo
> >> >> >>> Output queue: 0/40 (size/max)
> >> >> >>> 5 minute input rate 5381000 bits/sec, 2258 packets/sec
> >> >> >>> 5 minute output rate 15681000 bits/sec, 2623 packets/sec
> >> >> >>> 1271168625 packets input, 2747320900 bytes, 0 no buffer
> >> >> >>> Received 14588 broadcasts, 0 runts, 0 giants, 0 throttles
> >> >> >>> 36 input errors, 0 CRC, 0 frame, 0 overrun, 36 ignored
> >> >> >>> 0 watchdog
> >> >> >>> 0 input packets with dribble condition detected
> >> >> >>> 1476558580 packets output, 3223841169 bytes, 0 underruns
> >> >> >>> 0 output errors, 0 collisions, 0 interface resets
> >> >> >>> 0 babbles, 0 late collision, 0 deferred
> >> >> >>> 0 lost carrier, 0 no carrier
> >> >> >>> 0 output buffer failures, 0 output buffers swapped out
> >> >> >>>
> >> >> >>>
> >> >> >>> Nothing out of the ordinary.
> >> >> >>>
> >> >> >>>
> >> >> >>> Not quite sure how to proceed, I'm considering replacing the PA and/or
> >> >> >>> the VIP but if anybody can suggest this as being a s/w oddity I'm
> >> >> >>> willing to replace/upgrade as needed (bugtool is not turning up anything
> >> >> >>> of interest here)
> >> >> >>>
> >> >> >>>
> >> >> >>> Any help/advice on this muchly appreciated.
> >> >> >>>
> >> >> >>>
> >> >> >>> Thanks,
> >> >> >>>
> >> >> >>> Dave.
> >> >> >>>
> >> >> >>> _______________________________________________
> >> >> >>> cisco-nsp mailing list cisco-nsp at puck.nether.net
> >> >> >>> https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> >> >>> archive at http://puck.nether.net/pipermail/cisco-nsp/
> >> >> >> _______________________________________________
> >> >> >> cisco-nsp mailing list cisco-nsp at puck.nether.net
> >> >> >> https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> >> >> archive at http://puck.nether.net/pipermail/cisco-nsp/
> >> >> >>
> >> >> >
> >> >> > _______________________________________________
> >> >> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> >> >> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> >> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >> >> >
> >> >>
> >> >> _______________________________________________
> >> >> cisco-nsp mailing list cisco-nsp at puck.nether.net
> >> >> https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> >> archive at http://puck.nether.net/pipermail/cisco-nsp/
> >> > _______________________________________________
> >> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> >> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >> >
> >>
> >> _______________________________________________
> >> cisco-nsp mailing list cisco-nsp at puck.nether.net
> >> https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> archive at http://puck.nether.net/pipermail/cisco-nsp/
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list