[c-nsp] Cisco 7204VXR with NPE-G1 high CPU and output drops

Yap Chin Hoong - yapchinhoong at hotmail.com
Thu Apr 22 09:34:59 EDT 2010


Hi Youssef, kindly provide the output of the 'show proc cpu sorted' and 'show proc cpu history'. Thanks.
regards,YapCHhttp://itcertguides.blogspot.com/
> Date: Thu, 22 Apr 2010 15:06:11 +0200
> From: Youssef Bengelloun-Zahr <youssef at 720.fr>
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] Cisco 7204VXR with NPE-G1 high CPU and output drops
> Message-ID:
> 	<x2wcd86f9451004220606rd835ed28w86654612b4c76d69 at mail.gmail.com>
> Content-Type: text/plain; charset=windows-1252
> 
> Hello community,
> 
> I Have a Cisco 7204VXR router with NPE-G1 that started acting weird for 24
> hours. This router is dual attached to two 6k5 routers using multimode FO
> and SX GBICs.
> 
> The router is used as an LNS termination point for PPPoVPDN sessions, we
> have a bunch of them.
> 
> Here is a show ver output :
> 
> LNS1.IX1#sh version
> Cisco IOS Software, 7200 Software (C7200-ADVENTERPRISEK9-M), Version
> 12.2(33)SRD, RELEASE SOFTWARE (fc2)
> Technical Support: http://www.cisco.com/techsupport
> Copyright (c) 1986-2008 by Cisco Systems, Inc.
> Compiled Thu 23-Oct-08 12:58 by prod_rel_team
> 
> ROM: System Bootstrap, Version 12.3(4r)T1, RELEASE SOFTWARE (fc1)
> BOOTLDR: 7200 Software (C7200-KBOOT-M), Version 12.3(5a), RELEASE SOFTWARE
> (fc1)
> 
>  LNS1.IX1 uptime is 21 weeks, 23 hours, 22 minutes
> System returned to ROM by reload at 13:36:53 UTC Wed Nov 25 2009
> System restarted at 13:39:45 UTC Wed Nov 25 2009
> System image file is "disk0:c7200-adventerprisek9-mz.122-33.SRD.bin"
> Last reload type: Normal Reload
> Last reload reason: Reload command
> 
> 
> 
> This product contains cryptographic features and is subject to United
> States and local country laws governing import, export, transfer and
> use. Delivery of Cisco cryptographic products does not imply
> third-party authority to import, export, distribute or use encryption.
> Importers, exporters, distributors and users are responsible for
> compliance with U.S. and local country laws. By using this product you
> agree to comply with applicable laws and regulations. If you are unable
> to comply with U.S. and local laws, return this product immediately.
> 
> A summary of U.S. laws governing Cisco cryptographic products may be found
> at:
> http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
> 
> If you require further assistance please contact us by sending email to
> export at cisco.com.
> 
> Cisco 7204VXR (NPE-G1) processor (revision B) with 983040K/65536K bytes of
> memory.
> Processor board ID 29498611
> SB-1 CPU at 700Mhz, Implementation 0x401, Rev 0.2, 512KB L2 Cache
> 4 slot VXR midplane, Version 2.7
> 
> Last reset from power-on
> 
> PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points.
> Current configuration on bus mb1 has a total of 0 bandwidth points.
> This configuration is within the PCI bus capacity and is supported.
> 
> PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points.
> Current configuration on bus mb2 has a total of 0 bandwidth points.
> This configuration is within the PCI bus capacity and is supported.
> 
> Please refer to the following document "Cisco 7200 Series Port Adaptor
> Hardware Configuration Guidelines" on Cisco.com <http://www.cisco.com>
> for c7200 bandwidth points oversubscription and usage guidelines.
> 
> 
> 1 FastEthernet interface
> 3 Gigabit Ethernet interfaces
> 509K bytes of NVRAM.
> 
> 1000944K bytes of ATA PCMCIA card at slot 0 (Sector size 512 bytes).
> 1000944K bytes of ATA PCMCIA card at slot 1 (Sector size 512 bytes).
> 62592K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes).
> 16384K bytes of Flash internal SIMM (Sector size 256K).
> Configuration register is 0x2102
> 
> 
> Starting yesterday afternoon, I saw appear high CPU usage and numerous
> output drops. My first instinct was that GBICs started dying so I replaced,
> no change.
> 
> Then, I thought we were victim of a DDoS but my graphs show no increase of
> number of packets or things like that.
> 
> I have been debugging this and found out this :
> 
> LNS1.IX1#sh interfaces gi0/2 controller
> GigabitEthernet0/2 is up, line protocol is up
>   Hardware is BCM1250 Internal MAC, address is 000b.fcdd.c41a (bia
> 000b.fcdd.c41a)
>   Description: F=B, E=BB2.IX1, P=Gi9/1
>   Internet address is 77.246.80.101/31
>   MTU 9216 bytes, BW 1000000 Kbit, DLY 10 usec,
>      reliability 255/255, txload 29/255, rxload 34/255
>   Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
>   Keepalive set (10 sec)
>   Full Duplex, 1000Mbps, link type is auto, media type is SX
>   output flow-control is XON, input flow-control is XON
>   ARP type: ARPA, ARP Timeout 04:00:00
>   Last input 00:00:00, output 00:00:00, output hang never
>   Last clearing of "show interface" counters 00:22:35
>   Input queue: 1/150/0/0 (size/max/drops/flushes); Total output drops: 2712
>   *Queueing strategy: Class-based queueing*
>   Output queue: 311/1000/0 (size/max total/drops)
>   5 minute input rate 133959000 bits/sec, 28885 packets/sec
>   5 minute output rate 115013000 bits/sec, 20137 packets/sec
>      39283651 packets input, 1536817418 bytes, 0 no buffer
>      Received 535 broadcasts (0 IP multicasts)
>      0 runts, 0 giants, 0 throttles
>      164 input errors, 0 CRC, 0 frame, 164 overrun, 0 ignored
>      0 watchdog, 539 multicast, 0 pause input
>      27456129 packets output, 2625296538 bytes, 0 underruns
>      0 output errors, 0 collisions, 0 interface resets
>      0 babbles, 0 late collision, 0 deferred
>      0 lost carrier, 0 no carrier, 0 pause output
>      0 output buffer failures, 0 output buffers swapped out
> Interface GigabitEthernet0/2 (idb 0x50098218)
> Hardware is BCM1250 Internal MAC (Revision B2/B3)
>   Network connection mode is AUTO
>   network link is up
>   Config is 1000Mbps, Full Duplex
>   Selected media-type is GBIC
>   GBIC type is 1000BaseSX
>  MAC Registers:
>   mac_cfg =          0x000000C8000A0176, mac_thrsh_cfg = 0x0000080400084004
>   mac_vlantag =      0x0000000000000000, mac_frame_cfg = 0x241C400000280200
>   mac_adfilter_cfg = 0x0000000000000E28, mac_enable    = 0x0000000000000C11
>   mac_status =       0x0000000000040004, mac_int_mask  = 0x00004F0000C300C3
>   mac_txd_ctl =      0x000000000000000F, mac_eth_addr  = 0x00001AC4DDFC0B00
>   mac_fifo_ptrs =    0x241C400000280200, mac_eopcnt    = 0x000044001B1B1B1B
>   MAC RX is enabled  RX DMA - channel 0 is enabled, channel 1 is disabled
>   MAC TX is enabled  TX DMA - channel 0 is enabled, channel 1 is disabled
>   Device status = 1000 Mbps, Full-Duplex
>  PHY Registers:
>   PHY is Marvell 88E1011S (Rev 1.3)
>   Control                = 0x1000           Status                 = 0x796D
>   PHY ID 1               = 0x0141           PHY ID 2               = 0x0C62
>   Auto Neg Advertisement = 0x01A0           Link Partner Ability   = 0x4120
>   Auto Neg Expansion     = 0x0000           Next Page Tx           = 0x2001
>   Link Partner Next Page = 0x0000           1000BaseT Control      = 0x0000
>   1000BaseT Status       = 0x0000           Extended Status        = 0xC000
>   PHY Specific Control   = 0x0008           PHY Specific Status    = 0xAD04
>   Interrupt Enable       = 0x6C00           Interrupt Status       = 0x0000
>   Ext PHY Spec Control   = 0x0C64           Receive Error Counter  = 0x0000
>   LED Control            = 0x4100
>   Ext PHY Spec Control 2 = 0x006A           Ext PHY Spec Status    = 0xA017
>   PHY says Link is UP, Speed 1000Mbps, Full-Duplex [AUTONEG Done]
>   Physical Interface - GBIC
>   AUTONEG - Our ability is     1000M/FD Pause Capable (Asymmetric)
>   AUTONEG - Partner ability is 1000M/FD
>  GBIC registers:
>   Register 0x00:   01  04  01  00  00  00  01  20
>   Register 0x08:   40  0C  01  01  0D  00  00  00
>   Register 0x10:   37  1E  00  00  4F  45  4D  20
>   Register 0x18:   20  20  20  20  20  20  20  20
>   Register 0x20:   20  20  20  20  00  00  00  00
>   Register 0x28:   47  42  49  43  2D  53  58  20
>   Register 0x30:   20  20  20  20  20  20  20  20
>   Register 0x38:   00  00  00  00  03  52  00  BA
>   Register 0x40:   00  1A  00  00  42  31  30  34
>   Register 0x48:   38  31  39  31  20  20  20  20
>   Register 0x50:   20  20  20  20  30  39  30  39
>   Register 0x58:   32  39  20  20  68  B0  01  5A
>   Register 0x60:   20  20  20  20  20  20  20  20
>   Register 0x68:   20  20  20  20  20  20  20  20
>   Register 0x70:   20  20  20  20  20  20  20  20
>   Register 0x78:   20  20  20  20  20  20  20  20
>   PartNumber: GBIC-SX
>   PartRev: B
>   SerialNo: B1048191
>   Options:  0
>   Length(9um/50um/62.5um): 000/550/300
>   Date Code: 090929
>   Gigabit Ethernet Codes:  1
>  Internal Driver Information:
>   lc_ip_turbo_fs = 0x6236BE18, ip_routecache = 0x11 (dfs = 0/mdfs = 0)
>   rx cache size = 1000, rx cache end = 15
>   max_mtu = 9244
>  Software MAC address filter(hash:length/addr/mask/hits):
>  need_af_check = 0
>   0x00:  0  ffff.ffff.ffff  0000.0000.0000         0
>   0x2E:  0  0900.2b00.0005  0000.0000.0000         0
>   0x2F:  0  0900.2b00.0004  0000.0000.0000         0
>   0x5C:  0  0100.5e00.0002  0000.0000.0000         0
>   0xC0:  0  0100.0ccc.cccc  0000.0000.0000         0
>   0xD6:  0  0180.c200.0014  0000.0000.0000         0
>   0xD7:  0  0180.c200.0015  0000.0000.0000         0
>   0xE6:  0  000b.fcdd.c41a  0000.0000.0000         0
>   ring sizes: RX = 128, TX = 256
>   rx_particle_size: 512
>  Rx Channel 0:
>   dma_config0 =     0x0010002000800888, dma_config1 =    0x002D000000600029
>   dma_dscr_base =   0x000000000C218A40, dma_dscr_cnt =   0x0000000000000080
>   dma_cur_dscr_a =  0x000010000C29FC82, dma_cur_dscr_b = 0x02D4000000000001
>   dma_cur_daddr  =  0x000080000C2190E0
>   rxring = 0x0C218A40, shadow = 0x50098FA4, head = 26 (0x0C218BE0)
>   rx_overrun=78512, rx_nobuffer=0, rx_discard=0
>   Error Interrupts: rx_int_dscr = 0, rx_int_derr = 0, rx_int_drop = 53
>  Tx Channel 0:
>   dma_config0 =     0x0000000001001088, dma_config1 =    0x00B6000000000010
>   dma_dscr_base =   0x000000000C219280, dma_dscr_cnt =   0x0000000000000000
>   dma_cur_dscr_a =  0x00000F000C27F980, dma_cur_dscr_b = 0x0000000000000000
>   dma_cur_daddr  =  0x000000000C219430
>   txring = 0x0C219280, shadow = 0x657D0A14, head = 164, tail = 165, tx_count
> = 1
>   Error Interrupts: tx_int_dscr = 0, tx_int_derr = 0, tx_int_dzero = 0
>   chip_state = 2, ds->tx_limited = 0
>   throttled = 0, enabled = 0, disabled = 0
>   reset=6(init=1, restart=5), auto_restart=1
>   tx_underflow = 0, tx_overflow = 0
>   rx_underflow = 0, rx_overflow = 0, filtered_pak=0
>   descriptor mismatch = 0, fixed alignment = 52530
>   bad length = 0 dropped, 0 corrected
>   unexpected sop = 0
>  Address Filter:
>   Promiscuous mode OFF
>   Exact match table (for unicast, maximum 8 entries):
>     Entry 0 MAC Addr = 000b.fcdd.c41a
>     (All other entries are empty)
>   Hash match table (for multicast, maximum 8 entries):
>     Entry 0 MAC Addr = 0100.0ccc.cccc
>     Entry 1 MAC Addr = 0900.2b00.0004
>     Entry 2 MAC Addr = 0900.2b00.0005
>     Entry 3 MAC Addr = 0180.c200.0014
>     Entry 4 MAC Addr = 0180.c200.0015
>     Entry 5 MAC Addr = 0100.5e00.0002
>     (All other entries are empty)
>  Statistics:
>   Rx Bytes                 19767582054   Tx Bytes
> 19944011802
>   Rx Good Packets             27498925   Tx Good Packets
> 27480126
>   Rx Multicast                     544
>   Rx Broadcast                       0
> 
>   Rx Bad Pkt Errors                  0   Tx Bad Pkt Errors
> 0
>   Rx FCS Errors                      0   Tx FCS Errors
> 0
>   Rx Runt Errors                     0   Tx Runt Errors
> 0
>   Rx Oversize Errors                 0   Tx Oversize Errors
> 0
>   Rx Length Errors                   0   Tx Collisions
> 0
>   Rx Code Errors                     0   Tx Late Collisions
> 0
>   Rx Dribble Errors                  0   Tx Excessive Collisions
> 0
>                                          Tx Abort Errors
> 0
> 
> 
> My queuing strategy went from FIFO to Class-based queuing !?! How is that
> possible ?
> 
> Any ideas on what might be causing this ?
> 
> Thanks.
> 
> Regards.
> 
> Y.
> 
> -- 
> Youssef BENGELLOUN-ZAHR ??????????????????
> Ing?nieur R?seaux et T?l?coms
> 
> 
> Technopole de l'Aube  en Champagne - BP 601 - 10901 TROYES  Cedex 9
> Agence Paris : 6, rue Charles Floquet - 92120 MONTROUGE
> Tel                 +33 (0) 825 000 720
> Tel. direct      +33 (0) 1 77 35 59 14
> Tel. portable  +33 (0) 6 22 42 63 80
> Email            ybz at 720.fr
> ??????????????????????????????.....www.720.fr
> 
 		 	   		  
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4


More information about the cisco-nsp mailing list