[c-nsp] 7200 PA-GE interface resets
Beatty Lane-Davis
BLaneDavis at woosh.com
Tue Oct 19 20:04:28 EDT 2004
Hi all,
I've been having a problem with the PA-GE's in our 7200's for a while
now where they drop all sorts of packets under light load: input drops,
output drops, resets, flushes, ignoreds, overruns etc etc etc. I've had
a TAC case open for months now trying to figure out what's going on, but
that's gotten me both SFA and bitter, I must be asking the wrong
questions.
We found that by increasing the input and output hold queue's to the
maximum 4096(thanks Jared :-) _most_ of the symptoms went away, however,
we continued to see a constant trickle of ignoreds and resets on the two
boxes acting as our core routers. TAC has come back with a possible
explanation for the ignoreds, which I'll get to trying at some point...
My main point of concern at the moment has to do with the interface
resets. I've been advised by a TAC engineer that there is a known issue
with the PA-GE's where it resets periodically so I shouldn't be
concerned at all, that's just how it is. Right. Does anyone else have
this problem? He's identified two potential bugs that he's saying are
causing the resets, CSCdt37135 which don't look like it to me and
CSCdm84052 which I can't view in bugtracker. None of this has sounded
very right to me but with everything else we've got on it hasn't been
enough of a problem to get back to until yesterday afternoon when the
interface bounced on both the router and switch and a burst of resets
was incremented. Yikes!
I'm starting to wonder if there is some problem on the config between
the routers and switches? I understand from the colleague that built
this infrastructure that there was some tweaking that needed to be done
with regards to getting flow control et al right. Perhaps the
combination of parameters they hit upon are causing this problem now?
While these configs have been looked over more than a few times by
Cisco, I'm wondering if we're missing something basic and simple?
Configs and show's below. Is it possible that because the router is
having to ignore some of these packets that it's freaking out and
calling for an interface reset?
One key thing to keep in mind is that these routers should be really
bored, they're NPE-400's with no more than 60-70M mostly switched in and
out the same GE interface. Cricket shows that the CPU never runs hotter
than 25%.
Any advice or input would be very much appreciated.
Cheers,
b
ROUTER:
AKL-sky-core2#sho run int g2/0
Building configuration...
Current configuration : 215 bytes
!
interface GigabitEthernet2/0
description Link to Akl-Sky-Eth-Core2 Gig0/2
no ip address
no ip redirects
load-interval 30
negotiation auto
hold-queue 4096 in
hold-queue 4096 out
end
AKL-sky-core2#sho int g2/0
GigabitEthernet2/0 is up, line protocol is up
Hardware is WISEMAN, address is 0006.2ad6.bc38 (bia 0006.2ad6.bc38)
Description: Link to Akl-Sky-Eth-Core2 Gig0/2
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 16/255, rxload 17/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is autonegotiation, media type is SX
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 4w2d
Input queue: 0/4096/0/268 (size/max/drops/flushes); Total output
drops: 0
Queueing strategy: fifo
Output queue: 0/4096 (size/max)
30 second input rate 66940000 bits/sec, 18072 packets/sec
30 second output rate 64202000 bits/sec, 18332 packets/sec
2326211647 packets input, 129123951 bytes, 0 no buffer
Received 47941870 broadcasts, 0 runts, 2 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 17085 overrun, 1980368 ignored
0 watchdog, 27075463 multicast, 0 pause input
0 input packets with dribble condition detected
2768043357 packets output, 247793049 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
AKL-sky-core2#sho controllers g2/0
Interface GigabitEthernet2/0(idb 0x62E09644)
Hardware is WISEMAN 2.1, network connection mode is auto
network link is up
loopback type is none
startup time: 131969 usec
GBIC type is 1000BaseSX
idb->lc_ip_turbo_fs=0x605EF01C, ip_routecache=0x11(dfs=0/mdfs=0),
max_mtu=1528
fx1000_ds(tx)=0x62E0ADEC(0x62E0ADEC),
registers(tx)=0x3D000000(0x3D000000), curr_intr=0
rx cache size=2000, rx cache end=1857, rx_nobuffer=0
FX1000 registers:
CTRL =0x00000005, STATUS=0x0000000F
FCAL =0x00C28001, FCAH =0x00000100, FCT =0x00008808, FCTTV
=0x000016E3
RCTL =0x00428032, RDBAL0=0x0E5A9000, RDBAH0=0x00000000,
RDLEN0=0x00000800
RDH0 =0x00000008, RDT0 =0x00000007, RDTR0 =0x00000000, IMS
=0x000002D6
TCTL =0x000400FA, TIPG =0x00A0080A, TQC =0x00000000, TDBAL
=0x0E5AA000
TDBAH =0x00000000, TDLEN =0x00001000, TDH =0x00000075, TDT
=0x00000075
TXCW =0xC00001A0, RXCW =0xDC004060, FCRTL =0x80001200, FCRTH
=0x0000AFF0
RDFH =0x0000051F, RDFT =0x0000051F, TDFH =0x0000076A, TDFT
=0x0000076A
RX=normal, enabled TX=normal, enabled
Device status=full-duplex, link up, tx clock, rx clock
AN status=done(RF:0 , PAUSE:0 ), SYNC'ed, rx idle stream, rx invalid
symbols, rx idle char
PCI configuration registers:
bus_no=11, device_no=0
DeviceID=0x1000, VendorID=0x8086, Command=0x0116, Status=0x0200
Class=0x02/0x00/0x00, Revision=0x03, LatencyTimer=0xFC,
CacheLineSize=0x10
BaseAddr0=0x4D000004, BaseAddr1=0x00000000, MaxLat=0x00, MinGnt=0xFF
SubsysDeviceID=0x1000, SubsysVendorID=0x8086
Cap_Ptr=0x00000000 Retry/TRDY Timeout=0x00000000
PMC=0x00210001 PMCSR=0x00000000
FX1000(type=0x98) Internal Statistics:
rxring(128)=0xE5A9000, shadow=0x62E0B23C, head=7, rx_buf_size=512
txring(256)=0xE5AA000, shadow=0x62E0B468, head=116, tail=116
tx_int_txdw=0, tx_int_txqe=0, rx_int_rxdmt0=0, rx_int_rxt0=0
tx_count=0, txring_full=0, rx_max=0, filtered_pak=1182450
rx_overrun=0, rx_seq=0, reg_read=0, reg_write=0
rx_count=128, throttled=157, enabled=157, disabled=157
rx_no_enp=0, rx_discard=0, link_reset=0, pci_rev=3
tbl_overflow=0, chip_state=2, tx_nonint_done=0, tx_limited=0
reset=33(init=0, check=0, restart=32, pci=0), auto_restart=1
tx_carrier_loss=1, fatal_tx_err=0, tx_stucks_count=0
isl_err=0, wait_for_last_tdt=0, ctrl=18800005, ctrl0=800005
rx_stucks_count=6, rdtr_fpd=42455
HW addr filter: 0x62E0BC94, ISL disabled, Promiscuous mode multicast
Entry= 0: Addr=0006.2AD6.BC38
Entry= 1: Addr=FFFF.FFFF.FFFF
Entry= 2: Addr=0000.0000.0000
Entry= 3: Addr=0000.0000.0000
Entry= 4: Addr=0000.0000.0000
Entry= 5: Addr=0000.0000.0000
Entry= 6: Addr=0000.0000.0000
Entry= 7: Addr=0000.0000.0000
Entry= 8: Addr=0000.0000.0000
Entry= 9: Addr=0000.0000.0000
Entry=10: Addr=0000.0000.0000
Entry=11: Addr=0000.0000.0000
Entry=12: Addr=0000.0000.0000
Entry=13: Addr=0000.0000.0000
Entry=14: Addr=0000.0000.0000
Entry=15: Addr=0000.0000.0000
FX1000 Statistics (PA2)
CRC error 0 Symbol error 0
Missed Packets 17085 Single Collision 0
Excessive Coll 0 Multiple Coll 0
Late Coll 0 Collision 0
Defer 3743624539 Receive Length 0
Sequence Error 0 XON RX 0
XON TX 0 XOFF RX 0
XOFF TX 0 FC RX Unsupport 0
Packet RX (64) 16856309 Packet RX (127) 3768444986
Packet RX (255) 2553715898 Packet RX (511) 3425605101
Packet RX (1023) 1225869937 Packet RX (1522) 682745747
Good Packet RX 542120640 Broadcast RX 529757162
Multicast RX 275000715 Good Packet TX 0
Good Octets RX.H 9150 Good Octets RX.L 477895449
Good Octets TX.H 6544 Good Octets TX.L 135886086
RX No Buff 0 RX Undersize 0
RX Fragment 0 RX Oversize 0
RX Octets High 9154 RX Octets Low 2488501527
TX Octets High 6544 TX Octets Low 136030714
TX Packet 3035583678 RX Packet 1845821411
TX Broadcast 29709728 TX Multicast 18792709
Packet TX (64) 432407421 Packet TX (127) 2133793514
Packet TX (255) 1589796288 Packet TX (511) 3165520242
Packet TX (1023) 476549661 Packet TX (1522) 1437380480
AKL-sky-core2#sho diag 2
Slot 2:
Gigabit Ethernet Port adapter, 1 port
Port adapter is analyzed
Port adapter insertion time 1y15w ago
EEPROM contents at hardware discovery:
Hardware revision 1.0 Board revision B0
Serial number 25244843 Part number 73-3144-05
FRU Part Number: PA-1GE=
Test history 0x0 RMA number 00-00-00
EEPROM format version 1
EEPROM contents (hex):
0x20: 01 98 01 00 01 81 34 AB 49 0C 48 05 00 00 00 00
0x30: 58 00 00 00 00 00 00 00 00 01 FF FF FF FF FF FF
SWITCH:
interface GigabitEthernet0/2
description Trunk to akl-sky-core2.G2/0
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,2,9-12,221-224,817,818,1002-1005
switchport mode trunk
no ip address
flowcontrol send off
udld enable
end
akl-sky-eth-core2#sho int g0/2
GigabitEthernet0/2 is up, line protocol is up
Hardware is Gigabit Ethernet, address is 000c.85fe.3902 (bia
000c.85fe.3902)
Description: Trunk to akl-sky-core2.G2/0
MTU 1530 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 16/255, rxload 16/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is autonegotiation, media type is SX
output flow-control is off, input flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:40, output 00:00:00, output hang never
Last clearing of "show interface" counters 8w6d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue :0/40 (size/max)
5 minute input rate 64045000 bits/sec, 17930 packets/sec
5 minute output rate 66347000 bits/sec, 17667 packets/sec
659895699 packets input, 180512651 bytes, 0 no buffer
Received 11042264 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 4682489 multicast, 0 pause input
0 input packets with dribble condition detected
3469890038 packets output, 1981373737 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
akl-sky-eth-core2#sho controllers ethernet-controller gigabitEthernet
0/2
Transmit GigabitEthernet0/2 Receive
2621478988 Bytes 4113248337 Bytes
162381257 Unicast frames 3664271248 Unicast frames
275017802 Multicast frames 22614999 Multicast frames
127979814 Broadcast frames 35739595 Broadcast frames
0 Discarded frames 4192195 No dest, unicast
0 Too old frames 1330 No dest, multicast
0 Deferred frames 1091 No dest, broadcast
0 1 collision frames
0 2 collision frames 0 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 2091181826 Minimum size frames
0 8 collision frames 3490815167 65 to 127 byte frames
0 9 collision frames 1700243254 128 to 255 byte frames
0 10 collision frames 3634768013 256 to 511 byte frames
0 11 collision frames 475834251 512 to 1023 byte frames
0 12 collision frames 2490820455 1024 to 1518 byte frames
0 13 collision frames
0 14 collision frames 0 Flooded frames
0 15 collision frames 0 Overrun frames
0 Excessive collisions 11141 VLAN filtered frames
0 Late collisions 0 Source routed frames
0 Good (1 coll) frames 0 Valid oversize frames
0 Good(>1 coll) frames 0 Pause frames
0 Pause frames 0 Symbol error frames
0 VLAN discard frames 0 Invalid frames, too large
0 Excess defer frames 1960359057 Valid frames, too large
2163203250 Too large frames 0 Invalid frames, too small
2655676153 64 byte frames 767700323 Valid frames, too small
2427453886 127 byte frames
3533421271 255 byte frames
4147330143 511 byte frames
2274614786 1023 byte frames
543548568 1518 byte frames
More information about the cisco-nsp
mailing list