[c-nsp] c6k msfc - hsrp flapping

lee.e.rian at census.gov lee.e.rian at census.gov
Wed May 17 07:01:17 EDT 2006


We're having a problem on only one cat6000 MSFC where all the standby HSRP
interfaces go active and routing adjacencies drop.  It's like the packets
are being dropped somewhere inside the switch before they get to the MSFC.

The supervisor isn't complaining about any problems like links flapping or
spanning tree changes and none of the regular error counters on the
supervisor or MSFC look all that bad - but I did see some of the show ibc
counters incrementing.

I did a 'show ibc' before and after all the standby interfaces went active.
In less than 2 minutes resets went up by 5, missed packets by 1712, Defers
by 4 and RX No Buff by 366
Could this be why we're having problems?

msfc#sh ibc
Load for five secs: 9%/3%; one minute: 10%; five minutes: 7%
Time source is NTP, 15:49:43.638 EDT Tue May 16 2006

Interface IBC1/0(idb 0x6245FD6C)
Hardware is WISEMAN 1.5, network connection mode is auto
  network link is down
  loopback type is none
  startup time: 2101 usec
  GBIC is missing
  idb->lc_ip_turbo_fs=0x0, ip_routecache=0x0(dfs=0/mdfs=0), max_mtu=10286
  fx1000_ds(tx)=0x6246227C(0x6246227C),
registers(tx)=0x3C800000(0x3C800000), curr_intr=0
  rx cache size=1024, rx cache end=768, rx_nobuffer=0
 FX1000 registers:
  CTRL  =0x003C0005, STATUS=0x0000000F
  FCAL  =0x00C28001, FCAH  =0x00000100, FCT   =0x00008808, FCTTV
=0x00000080
  RCTL  =0x0040803A, RDBAL0=0x4B01D000, RDBAH0=0x00000000,
RDLEN0=0x00001000
  RDH0  =0x000000E9, RDT0  =0x000000E8, RDTR0 =0x00000000, IMS
=0x000002D5
  TCTL  =0x000400FA, TIPG  =0x00A0080A, TQC   =0x00000000, TDBAL
=0x4B01F000
  TDBAH =0x00000000, TDLEN =0x00002000, TDH   =0x0000006C, TDT
=0x0000006C
  TXCW  =0xC00001A0, RXCW  =0xDC004020, FCRTL =0x80001200, FCRTH
=0x0000AFF0
  RDFH  =0x0000088C, RDFT  =0x0000088C, TDFH  =0x0000048A, TDFT
=0x0000048A
  RX=normal, enabled  TX=normal, enabled
  Device status=full-duplex, link up, tx clock, rx clock
  AN status=done(RF:0 , PAUSE:0 ), SYNC'ed, rx idle stream, rx invalid
symbols, rx idle char
 PCI configuration registers:
  bus_no=1, device_no=1
  DeviceID=0x1000, VendorID=0x8086, Command=0x0116, Status=0x0200
  Class=0x02/0x00/0x00, Revision=0x03, LatencyTimer=0xFC,
CacheLineSize=0x10
  BaseAddr0=0x48800004, BaseAddr1=0x00000000, MaxLat=0x00, MinGnt=0xFF
  SubsysDeviceID=0x1000, SubsysVendorID=0x8086
  Cap_Ptr=0x00000000  Retry/TRDY Timeout=0x00000000
  PMC=0x00210001  PMCSR=0x00000000
 FX1000(type=0xB5) Internal Statistics:
  rxring(256)=0x4B01D000, shadow=0x6246271C, head=233, rx_buf_size=2048
  txring(512)=0x4B01F000, shadow=0x62462B48, head=108, tail=108
  tx_int_txdw=0, tx_int_txqe=0, rx_int_rxdmt0=0, rx_int_rxt0=0
  tx_count=0, txring_full=0, rx_max=0, filtered_pak=0
  rx_overrun=0, rx_seq=0, reg_read=0, reg_write=0
  rx_count=256, throttled=4, enabled=4, disabled=4
  rx_no_enp=0, rx_discard=0, link_reset=0, pci_rev=1
  tbl_overflow=0, chip_state=2, tx_nonint_done=0, tx_limited=0
  reset=46(init=0, check=0, restart=47, pci=0), auto_restart=0
  tx_carrier_loss=0, fatal_tx_err=0, tx_stucks_count=0
  isl_err=0, wait_for_last_tdt=0, ctrl=18800005, ctrl0=B40005
  rx_stucks_count=0, rdtr_fpd=11
 HW addr filter: 0x62463B74, ISL disabled, Promiscuous mode on
  Entry= 0:  Addr=00D0.C0D2.B990
  Entry= 1:  Addr=0000.0000.0000
  Entry= 2:  Addr=0000.0000.0000
  Entry= 3:  Addr=0000.0000.0000
  Entry= 4:  Addr=0000.0000.0000
  Entry= 5:  Addr=0000.0000.0000
  Entry= 6:  Addr=0000.0000.0000
  Entry= 7:  Addr=0000.0000.0000
  Entry= 8:  Addr=0000.0000.0000
  Entry= 9:  Addr=0000.0000.0000
  Entry=10:  Addr=0000.0000.0000
  Entry=11:  Addr=0000.0000.0000
  Entry=12:  Addr=0000.0000.0000
  Entry=13:  Addr=0000.0000.0000
  Entry=14:  Addr=0000.0000.0000
  Entry=15:  Addr=0000.0000.0000
FX1000 Statistics (PA1)
  CRC error        0             Symbol error     0
  Missed Packets   12330         Single Collision 0
  Excessive Coll   0             Multiple Coll    0
  Late Coll        0             Collision        0
  Defer            4             Receive Length   0
  Sequence Error   0             XON RX           0
  XON TX           0             XOFF RX          0
  XOFF TX          0             FC RX Unsupport  0
  Packet RX (64)   0             Packet RX (127)  296569
  Packet RX (255)  43039         Packet RX (511)  20033
  Packet RX (1023) 51935         Packet RX (1522) 314
  Good Packet RX   1832108       Broadcast RX     401678
  Multicast RX     0             Good Packet TX   0
  Good Octets RX.H 0             Good Octets RX.L 325981153
  Good Octets TX.H 0             Good Octets TX.L 234150677
  RX No Buff       3019          RX Undersize     0
  RX Fragment      0             RX Oversize      0
  RX Octets High   0             RX Octets Low    72559992
  TX Octets High   0             TX Octets Low    50774200
  TX Packet        209522        RX Packet        411890
  TX Broadcast     0             TX Multicast     1
  Packet TX (64)   0             Packet TX (127)  127673
  Packet TX (255)  13443         Packet TX (511)  12131
  Packet TX (1023) 55600         Packet TX (1522) 675

msfc#

 lots of
   %STANDBY-6-STATECHANGE: VlanXXX Group 1 state Standby -> Active
   %STANDBY-6-STATECHANGE: VlanXXX Group 1 state Standby -> Active
   %STANDBY-6-STATECHANGE: VlanXXX Group 1 state Standby -> Active
     . . .
   %STANDBY-6-STATECHANGE: VlanXXX Group 1 state Active -> Speak
   %STANDBY-6-STATECHANGE: VlanXXX Group 1 state Active -> Speak
   %STANDBY-6-STATECHANGE: VlanXXX Group 1 state Active -> Speak
 lines snipped

msfc#sh ibc
Load for five secs: 13%/4%; one minute: 12%; five minutes: 8%
Time source is NTP, 15:51:40.405 EDT Tue May 16 2006

Interface IBC1/0(idb 0x6245FD6C)
Hardware is WISEMAN 1.5, network connection mode is auto
  network link is down
  loopback type is none
  startup time: 2101 usec
  GBIC is missing
  idb->lc_ip_turbo_fs=0x0, ip_routecache=0x0(dfs=0/mdfs=0), max_mtu=10286
  fx1000_ds(tx)=0x6246227C(0x6246227C),
registers(tx)=0x3C800000(0x3C800000), curr_intr=0
  rx cache size=1024, rx cache end=768, rx_nobuffer=0
 FX1000 registers:
  CTRL  =0x003C0005, STATUS=0x0000000F
  FCAL  =0x00C28001, FCAH  =0x00000100, FCT   =0x00008808, FCTTV
=0x00000080
  RCTL  =0x0040803A, RDBAL0=0x4B01D000, RDBAH0=0x00000000,
RDLEN0=0x00001000
  RDH0  =0x000000EB, RDT0  =0x000000EA, RDTR0 =0x00000000, IMS
=0x000002D5
  TCTL  =0x000400FA, TIPG  =0x00A0080A, TQC   =0x00000000, TDBAL
=0x4B01F000
  TDBAH =0x00000000, TDLEN =0x00002000, TDH   =0x00000008, TDT
=0x00000008
  TXCW  =0xC00001A0, RXCW  =0xDC004020, FCRTL =0x80001200, FCRTH
=0x0000AFF0
  RDFH  =0x000012F7, RDFT  =0x000012F7, TDFH  =0x00000124, TDFT
=0x00000124
  RX=normal, enabled  TX=normal, enabled
  Device status=full-duplex, link up, tx clock, rx clock
  AN status=done(RF:0 , PAUSE:0 ), SYNC'ed, rx idle stream, rx invalid
symbols, rx idle char
 PCI configuration registers:
  bus_no=1, device_no=1
  DeviceID=0x1000, VendorID=0x8086, Command=0x0116, Status=0x0200
  Class=0x02/0x00/0x00, Revision=0x03, LatencyTimer=0xFC,
CacheLineSize=0x10
  BaseAddr0=0x48800004, BaseAddr1=0x00000000, MaxLat=0x00, MinGnt=0xFF
  SubsysDeviceID=0x1000, SubsysVendorID=0x8086
  Cap_Ptr=0x00000000  Retry/TRDY Timeout=0x00000000
  PMC=0x00210001  PMCSR=0x00000000
 FX1000(type=0xB5) Internal Statistics:
  rxring(256)=0x4B01D000, shadow=0x6246271C, head=235, rx_buf_size=2048
  txring(512)=0x4B01F000, shadow=0x62462B48, head=8, tail=8
  tx_int_txdw=0, tx_int_txqe=0, rx_int_rxdmt0=0, rx_int_rxt0=0
  tx_count=0, txring_full=0, rx_max=0, filtered_pak=0
  rx_overrun=0, rx_seq=0, reg_read=0, reg_write=0
  rx_count=256, throttled=4, enabled=4, disabled=4
  rx_no_enp=0, rx_discard=0, link_reset=0, pci_rev=1
  tbl_overflow=0, chip_state=2, tx_nonint_done=0, tx_limited=0
  reset=51(init=0, check=0, restart=52, pci=0), auto_restart=0
  tx_carrier_loss=0, fatal_tx_err=0, tx_stucks_count=0
  isl_err=0, wait_for_last_tdt=0, ctrl=18800005, ctrl0=B40005
  rx_stucks_count=0, rdtr_fpd=11
 HW addr filter: 0x62463B74, ISL disabled, Promiscuous mode on
  Entry= 0:  Addr=00D0.C0D2.B990
  Entry= 1:  Addr=0000.0000.0000
  Entry= 2:  Addr=0000.0000.0000
  Entry= 3:  Addr=0000.0000.0000
  Entry= 4:  Addr=0000.0000.0000
  Entry= 5:  Addr=0000.0000.0000
  Entry= 6:  Addr=0000.0000.0000
  Entry= 7:  Addr=0000.0000.0000
  Entry= 8:  Addr=0000.0000.0000
  Entry= 9:  Addr=0000.0000.0000
  Entry=10:  Addr=0000.0000.0000
  Entry=11:  Addr=0000.0000.0000
  Entry=12:  Addr=0000.0000.0000
  Entry=13:  Addr=0000.0000.0000
  Entry=14:  Addr=0000.0000.0000
  Entry=15:  Addr=0000.0000.0000
FX1000 Statistics (PA1)
  CRC error        0             Symbol error     0
  Missed Packets   14042         Single Collision 0
  Excessive Coll   0             Multiple Coll    0
  Late Coll        0             Collision        0
  Defer            8             Receive Length   0
  Sequence Error   0             XON RX           0
  XON TX           0             XOFF RX          0
  XOFF TX          0             FC RX Unsupport  0
  Packet RX (64)   0             Packet RX (127)  4716
  Packet RX (255)  851           Packet RX (511)  1036
  Packet RX (1023) 874           Packet RX (1522) 182
  Good Packet RX   1839767       Broadcast RX     5246
  Multicast RX     0             Good Packet TX   0
  Good Octets RX.H 0             Good Octets RX.L 327651437
  Good Octets TX.H 0             Good Octets TX.L 236717870
  RX No Buff       3385          RX Undersize     0
  RX Fragment      0             RX Oversize      0
  RX Octets High   0             RX Octets Low    1670284
  TX Octets High   0             TX Octets Low    2567193
  TX Packet        7181          RX Packet        7659
  TX Broadcast     0             TX Multicast     1
  Packet TX (64)   0             Packet TX (127)  2413
  Packet TX (255)  422           Packet TX (511)  1370
  Packet TX (1023) 2765          Packet TX (1522) 211

msfc#

Thanks,
Lee



More information about the cisco-nsp mailing list