[c-nsp] ME3400 high cpu

CWO Network Operations network at cwo.com
Fri Sep 20 13:52:33 EDT 2013


Hello,

We just started using the ME3400's (ME-3400G-12CS-A, 12.2(58)SE2, ME340x-METROIPACCESSK9-M ) in our network and are noticing a fairly high CPU load compared what we were used to on the older 3550's.
The config is pretty much the same on the new M#3400's then what we had on the 3550's. The 3550's never really ran higher than 3%/0% CPU load.
So, I'm not sure why the ME3400's are running so much higher in CPU... 

Here are some outputs:

switch # sho proc cpu sorted 5sec
PU utilization for five seconds: 30%/20%; one minute: 34%; five minutes: 32%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  85      186523       11652      16007  0.79%  0.15%  0.12%   0 HULC Tcam Memory 
  12      402028     1335248        301  0.63%  0.55%  0.57%   0 ARP Input        
  96       91391     1707572         53  0.47%  0.13%  0.11%   0 hpm main process 
  62       63815        2667      23927  0.47%  0.05%  0.00%   0 Per-minute Jobs  
 200       59364      425347        139  0.31%  0.07%  0.05%   0 Spanning Tree    
  68      100263     6213238         16  0.31%  0.15%  0.10%   0 Fifo Error Detec 
 192      139880      328694        425  0.31%  0.20%  0.16%   0 IP Input         
 133       92952      206477        450  0.31%  0.19%  0.15%   0 HL3U bkgrd proce 
 316       61245      174270        351  0.15%  0.04%  0.05%   0 IP SNMP          
 113        6308      579307         10  0.15%  0.01%  0.00%   0 Hulc Storm Contr 
  83       17874     3460721          5  0.15%  0.05%  0.01%   0 HLFM address ret 
 142       22760       46634        488  0.15%  0.03%  0.00%   0 HRPC qos request 
 198      110177      244449        450  0.15%  0.19%  0.16%   0 ADJ resolve proc 
 318      103809       87120       1191  0.15%  0.10%  0.09%   0 SNMP ENGINE      
  14           0           1          0  0.00%  0.00%  0.00%   0 CEF MIB API      
  13       17713      121776        145  0.00%  0.03%  0.00%   0 ARP Background   
   5          82         106        773  0.00%  0.00%  0.00%   0 Pool Manager     
 [...]

It looks like we are punting packets like crazy...


switch# show platform port-asic stats drop
Port-asic Port Drop Statistics - Summary
========================================
  RxQueue 0 Drop Stats: 0
  RxQueue 1 Drop Stats: 0
  RxQueue 2 Drop Stats: 0
  RxQueue 3 Drop Stats: 0

  Port  0 TxQueue Drop Stats: 0
  Port  1 TxQueue Drop Stats: 0
  Port  2 TxQueue Drop Stats: 0
  Port  3 TxQueue Drop Stats: 0

  Supervisor TxQueue Drop Statistics
    Queue  0: 0
    Queue  1: 0
    Queue  2: 0
    Queue  3: 0
    Queue  4: 0
    Queue  5: 0
    Queue  6: 17931
    Queue  7: 0
    Queue  8: 82
    Queue  9: 0
    Queue 10: 0
    Queue 11: 46734696
    Queue 12: 0
    Queue 13: 0
    Queue 14: 0
    Queue 15: 0

Port-asic Port Drop Statistics - Details
========================================
  RxQueue Drop Statistics
    Queue 0
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 1
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 2
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 3
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0

  Port 0 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 1 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 2 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0

  Port 3 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
  Supervisor TxQueue Drop Statistics
    Queue  0: 0
    Queue  1: 0
    Queue  2: 0
    Queue  3: 0
    Queue  4: 0
    Queue  5: 0
    Queue  6: 17931
    Queue  7: 0
    Queue  8: 82
    Queue  9: 0
    Queue 10: 0
    Queue 11: 46734702
    Queue 12: 0
    Queue 13: 0
    Queue 14: 0
    Queue 15: 0

switch# show controllers cpu-interface 
ASIC    Rxbiterr   Rxunder    Fwdctfix   Txbuflos   Rxbufloc   Rxbufdrain
-------------------------------------------------------------------------
ASIC0     0          0          0          0          0          0         
ASIC1     0          0          0          0          0          0         
ASIC2     0          0          0          0          0          0         
ASIC3     0          0          0          0          0          0         

HOL Fix Counts
--------------
No Fixes:          0 Added:          4 In Use:          0 Both:          0

CPU Heartbeat Statistics

Tx Success Tx Fail    1st Thr    2nd Thr    Unthr      RetryCtMax
---------- ---------- ---------- ---------- ---------- ----------
   2320747          0          0          0          0          1

Rx Delay
         0          1          2          3          4 
---------- ---------- ---------- ---------- ---------- 
   2320747          0          0          0          0 

AddlDelay AdvanceCnt
---------- ----------
         0          0

Rx Retries by RetryCount
         0          1          2          3          4          5          6 
---------- ---------- ---------- ---------- ---------- ---------- ---------- 
   2320747          0          0          0          0          0          0 

         7          8          9 
---------- ---------- ---------- 
         0          0          0 

AddlRetry
----------
         0

          
cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc               0          0          0          0          0         
stp               234154     0          0          0          0         
ipc               0          0          0          0          0         
routing protocol  1871818    0          0          0          0         
L2 protocol       208159     0          0          0          0         
remote console    0          0          0          0          0         
sw forwarding     14959846   0          0          0          0         
host              305132     0          0          0          0         
broadcast         1754197    0          0          0          0         
cbt-to-spt        0          0          0          0          0         
igmp snooping     369150     0          0          0          0         
icmp              731133339  0          0          4          0         
logging           0          0          0          0          0         
rpf-fail          0          0          0          0          0         
dstats            0          0          0          0          0         
cpu heartbeat     2320747    0          0          0          0         

cpu-queue         static inuse static added
----------------- ------------ ------------
rpc               0            0           
stp               0            0           
ipc               0            0           
routing protocol  0            0           
L2 protocol       0            0           
remote console    0            0           
sw forwarding     0            0           
host              0            0           
broadcast         0            0           
cbt-to-spt        0            0           
igmp snooping     0            0           
icmp              0            4           
logging           0            0           
rpf-fail          0            0           
dstats            0            0           
cpu heartbeat     0            0           

Supervisor ASIC receive-queue parameters
----------------------------------------
 queue 0 maxrecevsize 7E0 pakhead 503C76C paktail 5E93634
 queue 1 maxrecevsize 7E0 pakhead 5133F00 paktail 5133A7C
 queue 2 maxrecevsize 7E0 pakhead 50F79B4 paktail 5F1231C
 queue 3 maxrecevsize 7E0 pakhead 525CA38 paktail 5261278
 queue 4 maxrecevsize 7E0 pakhead 513E7A0 paktail 5F4CF30
 queue 5 maxrecevsize 7E0 pakhead 522EF20 paktail 606BA9C
 queue 6 maxrecevsize 7E0 pakhead 60857C8 paktail 6085344
 queue 7 maxrecevsize 7E0 pakhead 5FF3E9C paktail 5211B7C
 queue 8 maxrecevsize 7E0 pakhead 602838C paktail 602B538
 queue 9 maxrecevsize 7E0 pakhead 5FC16C8 paktail 5FC16C8
 queue A maxrecevsize 7E0 pakhead 5171904 paktail 5F824F4
 queue B maxrecevsize 7E0 pakhead 526896C paktail 5268DF0
 queue C maxrecevsize 7E0 pakhead 5178C54 paktail 5F97AC0
 queue D maxrecevsize 7E0 pakhead 4E04E74 paktail 51B556C
 queue E maxrecevsize 0 pakhead 0 paktail 0
 queue F maxrecevsize 7E0 pakhead 516D9CC paktail 5F7E8B4
Supervisor ASIC exception status
--------------------------------
Receive overrun    00000000   Transmit overrun 00000000
FrameSignatureErr  00000000   MicInitialize    00000000
BadFrameErr        00000000   LenExceededErr   00000000
BadJumboSegments   00000000

Supervisor ASIC Mic Registers
------------------------------
MicDirectPollInfo     		80000200
MicIndicationsReceived		00000000
MicInterruptsReceived		00000000
MicPcsInfo           		0000001F
MicPlbMasterConfiguration	00000000
MicRxFifosAvailable  		00000000
MicRxFifosReady      		0000B7FF
MicTimeOutPeriod:	FrameTOPeriod: 00000EA6	DirectTOPeriod: 00004000
MicTransmFramesCopied		00000003
MicTxFifosAvailable  		0000000F
MicConfiguration:	Conf flag: 00000110	Interrupt Flag: 0000000A
MicReceiveFifoAssignmen	Queue 0 - 7: 00000000	Queue 8 - 15:00000000
MicReceiveFramesReady:	FrameAvailable: 00000880	frameAvaiMask: 00000000
MicException:    
        Exception_flag	00000000
	Message-1	00000000
	Message-2	00000000
	Message-3	00000000
MicIntRxFifo:
	ReadPtr		000000D8	WritePtr	000000D8
	WHeadPtr	000000D8	TxFifoDepth	C0000800
MicIntTxFifo:
	ReadPtr		000008C8	WritePtr	000008C8
	WHeadPtr	000008C8	TxFifoDepth	C0000800
MicDecodeInfo:
Fifo0:	address:	03FF4000 asic_num:	00000100
Fifo1:	address:	03FF4400 asic_num:	00000101
MicTransmitFifoInfo:
Fifo0:	 StartPtrs:	066CE800	ReadPtr:	066CEDE8
	WritePtrs:	066CEDE8	Fifo_Flag:	8A800800
	Weights:	001E001E
Fifo1:	 StartPtrs:	0642D000	ReadPtr:	0642D018
	WritePtrs:	0642D018	Fifo_Flag:	89800400
	Weights:	000A000A
MicReceiveFifoInfo:
Fifo0:	StartPtr:	066E2000	ReadPtr:	066E2FA0
	WritePtrs:	066E2000	Fifo_Flag:	8B000060
	writeHeaderPtr:	066E2000
Fifo1:	StartPtr:	069BF000	ReadPtr:	069BF158
	WritePtrs:	069BF158	Fifo_Flag:	89800400
	writeHeaderPtr:	069BF158
Fifo2:	StartPtr:	06977800	ReadPtr:	06977800
	WritePtrs:	06977800	Fifo_Flag:	89800400
	writeHeaderPtr:	06977800
Fifo3:	StartPtr:	06C80000	ReadPtr:	06C80250
	WritePtrs:	06C80250	Fifo_Flag:	89800400
	writeHeaderPtr:	06C80250
Fifo4:	StartPtr:	06A06800	ReadPtr:	06A068F8
	WritePtrs:	06A068F8	Fifo_Flag:	89800400
	writeHeaderPtr:	06A068F8
Fifo5:	StartPtr:	06BEF800	ReadPtr:	06BEF800
	WritePtrs:	06BEF800	Fifo_Flag:	88800200
	writeHeaderPtr:	06BEF800
Fifo6:	StartPtr:	06C36400	ReadPtr:	06C36748
        WritePtrs:	06C36748	Fifo_Flag:	89800400
	writeHeaderPtr:	06C36748
Fifo7:	StartPtr:	06AB3000	ReadPtr:	06AB39B0
	WritePtrs:	06AB3F60	Fifo_Flag:	8B000A50
	writeHeaderPtr:	06AB3F60
Fifo8:	StartPtr:	06BC9800	ReadPtr:	06BC98A8
	WritePtrs:	06BC98A8	Fifo_Flag:	88800200
	writeHeaderPtr:	06BC98A8
Fifo9:	StartPtr:	066E0CD8	ReadPtr:	066E0CD8
	WritePtrs:	066E0CD8	Fifo_Flag:	82800008
	writeHeaderPtr:	066E0CD8
Fifo10:	StartPtr:	06A50000	ReadPtr:	06A501F0
	WritePtrs:	06A501F0	Fifo_Flag:	88800200
	writeHeaderPtr:	06A501F0
Fifo11:	StartPtr:	06C36300	ReadPtr:	06C36378
	WritePtrs:	06C36378	Fifo_Flag:	86400000
	writeHeaderPtr:	06C36378
Fifo12:	StartPtr:	06A73C00	ReadPtr:	06A73F00
	WritePtrs:	06A73C00	Fifo_Flag:	89000100
	writeHeaderPtr:	06A73C00
Fifo13:	StartPtr:	0642CF00	ReadPtr:	0642CF00
	WritePtrs:	0642CF00	Fifo_Flag:	86800080
	writeHeaderPtr:	0642CF00
Fifo14:	StartPtr:	00000000	ReadPtr:	00000000
	WritePtrs:	00000000	Fifo_Flag:	00800000
	writeHeaderPtr:	00000000
Fifo15:	StartPtr:	0642D480	ReadPtr:	0642D498
	WritePtrs:	0642D498	Fifo_Flag:	84800020
	writeHeaderPtr:	0642D498


===========================================================
Complete Board Id:0x0060
===========================================================


We are using the default template...


switch# (sho sdm prefer 
 The current template is "default" template.
 The selected template optimizes the resources in
 the switch to support this level of features for
 8 routed interfaces and 1024 VLANs. 

  number of unicast mac addresses:                  5K
  number of IPv4 IGMP groups + multicast routes:    1K
  number of IPv4 unicast routes:                    9K
    number of directly-connected IPv4 hosts:        5K
    number of indirect IPv4 routes:                 4K
  number of IPv4 policy based routing aces:         0.5K
  number of IPv4/MAC qos aces:                      0.5K
  number of IPv4/MAC security aces:                 1K



switch# sho ip cef sum
IPv4 CEF is enabled for distributed and running
VRF Default
 457 prefixes (457/0 fwd/non-fwd)
 Table id 0x0
 Database epoch:        2 (457 entries at this epoch)


switch# sho ip route sum
IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source    Networks    Subnets     Replicates  Overhead    Memory (bytes)
connected       0           10          0           600         1720
static          4           22          0           1560        4472
ospf 1          1           148         0           8940        26224
  Intra-area: 79 Inter-area: 0 External-1: 43 External-2: 27
  NSSA External-1: 0 NSSA External-2: 0
internal        5                                               9288
Total           10          180         0           11100       41704


Do you see anything obvious that may cause the high CPU load?

Thanks for your help

JB




More information about the cisco-nsp mailing list