[c-nsp] Huge SP CPU usage spikes 100%
james list
jameslist72 at gmail.com
Thu Mar 1 04:53:58 EST 2018
Here the output, I can share netcapture privately if you agree.
Thanks
xxx#show platform netint
Network IO Interrupt Throttling:
throttle count=122323 (0 in multicast fs), timer count=122323
active=0, configured=1
netint usec=4000, netint mask usec=800
resource netint mask usec=400
inband_throttle_mask_hi = 0x0
inband_throttle_mask_lo = 0x800000
SPD-drop triggered throttles=265
SPD triggered unthrottles=192
Max SPD throttle time=4000 usecs
Buffer depletion triggered throttles=0
ARP queue triggered throttles=0
Low inband activity:throttles=0, active=0, min=5 pkts/6000 bytes, 1000 usec
Low activity threshold 1500 pps (192)
Current rx packet rate 742 pps (95)
Idle hook unthrottles=89404
xxx#remote command switch show platform hardware earl status
Superman 0 interrupt counts : Total 6912472122
se_one: 0
se_hwm: 10653
se_dn: 280184388
ntfy_one: 0
ntfy_hwm: 0
ntfy_ovfl: 0
ft_b0_corr_ecc: 0
ft_b0_uncorr_ecc: 0
ft_b1_corr_ecc: 0
ft_b1_uncorr_ecc: 0
ft_b0_multi_pg_hit: 0
ft_b1_multi_pg_hit: 0
l2_flush_done: 0
loop_ntfy: 0
ntfy_fifo_full: 0
l2_line_full: 0
mc_cap: 6707068077
b0_invld_pg_acc: 0
b1_invld_pg_acc: 0
pkt_crc32_err: 0
cpu_par_err: 0
cpu_timeout: 0
rbus_timeout: 0
dbus_timeout: 0
ip_chksum_err: 0
l3_fcs_err: 0
dbus_seq_err: 0
dbus_hdr_err: 0
l2l3_seq_mismatch: 0
l3_rcv_ovfl_err: 0
l2_merge_err: 0
============================
Tycho Interrupts: Total - 320837111
CPU interrupt block : Total interrupts - 0
Netflow interrupt block : Total interrupts - 0
IFIFO_OVF_INT : 0
SINGLE_BIT_ECC_INT : 0
MULTI_BIT_ECC_INT : 0
ECC_DATA_CAPT_INT : 0
TCAM_THRLD_EXCD_INT : 0
ICAM_THRLD_EXCD_INT : 0
TABLE_FULL_INT : 0
ENTRY_ALIAS_INT : 0
FIB interrupt block : Total interrupts - 0
FC_FOVR_INT : 0
FC_FUDR_INT : 0
BP_FOVR_INT : 0
BP_FUDR_INT : 0
RS_FOVR_INT : 0
RS_FUDR_INT : 0
AJ_FOVR_INT : 0
AJ_FUDR_INT : 0
FB_COR_ECC_INT : 0
FB_UNCOR_ECC_INT : 0
Search interrupt block : Total interrupts - 320833986
Adj. Statistics tabl block : Total interrupts - 0
Adj. table interface block : Total interrupts - 0
AT_SEQ_ERR_INT : 0
AT_FOVR_INT : 0
AT_FUDR_INT : 0
AT_IB_ADJ_INT : 0
AT_BZONE_INT : 0
AT_CORR_ECC_ERR_INT : 0
AT_UNCORR_ECC_ERR_INT : 0
AT_ECC_ERR_DATA_CAPT : 0
Packet Parser block block : Total interrupts - 1881
IP_LEN_INT : 1881
IP_SHRT_INT : 0
IP_CS_INT : 0
QDBUS_CRC_INT : 0
QDBUS_LEN_INT : 0
IB_LEN_INT : 0
IB_SEC_INT : 0
Decision Engine bloc block : Total interrupts - 0
NS_ECC_SBE_INT : 0
NS_ECC_MBE_INT : 0
NS_ECC_DCE_INT : 0
ACL_SEQ_ERR_INT : 0
STAT_FOVF_INT : 0
RSLT_FOVF_INT : 0
GEM_MEM_LEAK_INT : 0
Rewrite block interr block : Total interrupts - 0
RW_VO_INT : 0
RW_VU_INT : 0
RW_RO_INT : 0
RW_RU_INT : 0
Statistics block int block : Total interrupts - 1245
GLOBAL_OVFL : 0
BGP_OVFL : 1245
VLAN_OVFL_0 : 0
VLAN_OVFL_1 : 0
VLAN_OVFL_2 : 0
VLAN_OVFL_3 : 0
VLAN_OVFL_4 : 0
VLAN_OVFL_5 : 0
INIT_DONE : 0
Level 4 Map interfac block : Total interrupts - 0
L4MAP_OVFL : 0
Key Queue Block inte block : Total interrupts - 0
KQ_OVFL : 0
Interrupt statistics for Kuma 0
--------------------------------
Kuma cpu parity error : 0
Kuma K interface: data intf crc error : 0
Kuma K'interface: data intf crc error : 0
Kuma K interface: result intf hdr fcs err : 0
Kuma K interface: result intf data fcs err : 0
Kuma K'interface: result intf hdr fcs err : 0
Kuma K'interface: result intf data fcs err : 0
Kuma K interface: intf freeze err : 0
Kuma K'interface: intf freeze err : 0
Kuma stats overflow_0 err : 0
Kuma stats overflow_1 err : 0
Kuma stats saturate_0 err : 0
Kuma stats saturate_1 err : 0
Kuma K'interface err : 0
Kuma K interface err : 0
Kuma E interface: frame crc error on received dbus frame : 0
Kuma E interface: header crc error on received dbus frame : 0
Kuma E interface: length error on received dbus frame : 0
Kuma E interface: received rbus frame seqeunce mismatch err : 0
Kuma E interface: seqtbl overflow err : 0
Kuma E interface: seqtbl underrun err : 0
Kuma E interface: seqtbl is not empty when dbus or rbus out : 0
Kuma E interface: rbrx fcs1/header checksum err : 0
Kuma E interface: rbrx fcs2/data checksum err : 0
Kuma E interface: dbus seq num out of order err : 0
Kuma E interface: rbus seq num out of order : 0
Kuma E interface: dbus fci counter reached : 0
Kuma E interface: dbus idle timeout : 0
Kuma E interface: rbus idle timeout : 0
Kuma E interface: rbus async fifo flow controlled : 0
Kuma E interface: dbus async fifo flow controlled : 0
Kuma E interface: dbus transmit for K' fifo full : 0
Kuma E interface: dbus transmit for K fifo full : 0
Kuma E interface: rbus transmit fifo full : 0
Kuma E interface: rbus transmit fcs1 err : 0
Kuma E interface: rbus transmit fcs2 err : 0
No of times all interrupts got throttled : 0
No of times all interrupts got unthrottled : 0
Kuma 1 Soft Reset Reg
SOFT_RESET_REG: FFFF
Kuma 1 EIF Ebus Cfg Reg
EIF_EBUS_CFG: E9C9Forwarding statistics for kuma 0
---------------------------------
KIF DBUS PKTS: 0x0000008A30E10EE1
KIF RBUS PKTS: 0x0000008A30E10EE0
KIF DBUS PKT CRC ERRORS: 0x0000000000000000
KIF RBUS PKT HDR FCS ERRORS: 0x0000000000000000
KIF RBUS PKT DATA ERRORS: 0x0000000000000000
PIF RBUS PKTS: 0x0000008A30E10ECF
PIF DBUS PKT CRC ERRORS: 0x0000000000000000
PIF RBUS PKT HDR FCS ERRORS: 0x0000000000000000
PIF RBUS PKT DATA ERRORS: 0x0000000000000000
EIF DBUS COMPACT PKTS: 0x0000000000000000
EIF DBUS LEGACY PKTS: 0x0000000000000000
EIF DBUS TRUNCATED PKTS: 0x0000000000000000
EIF DBUS EXTENDED PKTS: 0x0000000000000000
EIF RBUS NORMAL PKTS: 0x0000000000000000
EIF RBUS EXTENDED PKTS: 0x0000000000000000
EIF DBUS PKT HDR CRC ERRORS: 0x0000000000000000
EIF DBUS PKT DATA CRC ERRORS: 0x0000000000000000
EIF RBUS PKT HDR FCS ERRORS: 0x0000000000000000
EIF RBUS PKT DATA ERRORS: 0x0000000000000000
xxx#show ibc
Interface information:
Interface IBC0/0(idb 0x4809CED4)
Hardware is Mistral IBC (revision 5)
5 minute rx rate 944000 bits/sec, 793 packets/sec
5 minute tx rate 25000 bits/sec, 37 packets/sec
62536015398 packets input, 10365891883465 bytes
0 broadcasts received
7027746331 packets output, 621038630448 bytes
2101396473 broadcasts sent
14 Inband input packet drops
0 Bridge Packet loopback drops
0 Rx packets dropped with Multicast MAC and Unicast IP
2467023087 Packets out of 554699386 CEF Switched, 0 Packets out of
0 Tag CEF Switched
3916625157 Packets Fast Switched
0 Giant Tx packets dropped
0 Packets SLB Switched, 0 Packets CWAN Switched
Potential/Actual paks copied to process level 228808364/225833216
(2975148 dropped, 265 spd drops)
194443558 paks copied at vlan soutput
1163732006 paks copied at ibc fast soutput
0 paks copied at ibc fastsend
296471304 inband interrupts
176660714 transmit ring cleanups
121418594 ibl inputs
176660714 total tx interrupts set
176660714 tx ints due to packets outstanding
0 tx ints due low free buffers in pool
0 tx ints due to application setting
tx dma done batch size=32
buffers free minimum before tx int=4
mistral ran out of tx descriptors 0 times
mistral tx interrupt inconsisteny occured 0 times
Label switched pkts dropped: 0
Xconnect pkts processed: 0, dropped: 0
Xconnect pkt reflection drops: 0
IBC resets = 2; last at 03:09:59.964 CET Mon Jan 30 2012
MISTRAL ERROR COUNTERS
System address timeouts = 0 BUS errors = 0
IBC Address timeouts = 0 (addr 0x0)
Page CRC errors = 0 IBL CRC errors = 0
ECC Correctable errors = 0
Packets with padding removed (6048/0/0) = 6048
Packets expanded (0/0) = 0
Packets attempted tail end expansion > 1 page and were dropped = 0
IP packets dropped with frag offset of 1 = 0
2974883 total packets dropped on throttled interfaces (2954630 low,
16704 medium, 3549 high)
0 medium, 0 high prio packets allowed in due to selective throttling
0 packets corrected for if_input
0 packets were uncorrectable for if_input
0 exception packets dropped with service acceleration enabled
Hazard Illegal packet length = 0 Illegal Offset = 0
Hazard Packet underflow = 0 Packet Overflow = 0
IBL fill hang count = 0 Unencapsed packets = 0
LBIC RXQ Drop pkt count = 0 LBIC drop pkt count = 0
LBIC Drop pkt stick = 0
DMA Instance Details:
DMA Instance = 0x4809DD70 RX Pak = 0x47D219B0
Receive Ring = 0x092209C0 Receive Shadow = 0x50BA9CC4
Receive Tail = 0 Receive Head = 0
Receive Count = 0
Transmit Ring = 0x09222A00 Transmit Shadow = 0x4809DE4C
Transmit Tail = 333 Transmit Head = 320
Transmit Count = 13
MISTRAL REGISTERS:
soft_reset_cfg = 0x00040000 gmac_status = 0x1
MISC GLOBAL REGISTERS:
global_cfg = 0x00000020 mis_init_sts = 0xF
tm_init_size_cfg = 0x00008000 sysad_timeout_cfg = 0xFF
dimm_parm_cfg_hi = 0x00000576 dimm_parm_cfg_lo = 0x42040F5A
FAST SWITCHING ASSISTANT HARDWARE REGISTERS:
send_router_cfg = 0x00000380 send_bridge_cfg = 0x0387
ibl_cfg = 0xF
obl_sem_req = 0x00000000 obl_sem_sts = 0x0
INBAND DMA REGISTERS:
rxdma: sts = 0x00000000 base = 0xD2209C0
rxdscp_cnt = 0
txdma: sts = 0x00000000 base = 0xD223558
txdscp_cnt = 0
dma_buffer_size = 0x3
BUFFER ALLOCATION, WRR and STATUS REGISTERS:
lbic_pkt_type[0] = 0x76 lbic_pkt_type[1] = 0x74
lbic_pkt_type[2] = 0x74 lbic_pkt_type[3] = 0x74
lbic_pkt_type[4] = 0x74 lbic_pkt_type[5] = 0x74
lbic_pkt_type[6] = 0x74 lbic_pkt_type[7] = 0x76
idx_direc_rx_offset = 0x76
rq_wrr_ratio = 0x0
INTERRUPTS:
int_mask_hi = 0x0000007E int_mask_lo =
0xE7001A58
int_source_status_hi = 0x00000000 int_source_status_lo =
0x00001000
int_status_0_hi = 0x00000000 int_status_0_lo =
0x00000000
int_status_1_hi = 0x00000000 int_status_1_lo =
0x00000000
int_status_2_hi = 0x00000000 int_status_2_lo =
0x00000000
int_status_3_hi = 0x00000000 int_status_3_lo =
0x00000000
int_status_4_hi = 0x00000000 int_status_4_lo =
0x00000000
map[00] = 0xA map[01] = 0x2 map[02] = 0x2
map[03] = 0x0
map[04] = 0x0 map[05] = 0x3 map[06] = 0x3
map[07] = 0x2
map[08] = 0x1 map[09] = 0x1 map[10] = 0x1
map[11] = 0x0
map[12] = 0x0 map[13] = 0x0 map[14] = 0x4
map[15] = 0x4
map[16] = 0x4 map[17] = 0x4 map[18] = 0x4
map[19] = 0x4
map[20] = 0x4 map[21] = 0x4 map[22] = 0x4
map[23] = 0x0
map[24] = 0x1 map[25] = 0x1 map[26] = 0x1
map[27] = 0x4
map[28] = 0x4 map[29] = 0x0 map[30] = 0x0
map[31] = 0x0
map[32] = 0x4 map[33] = 0x0 map[34] = 0x0
map[35] = 0x0
map[36] = 0x0 map[37] = 0x0 map[38] = 0x0
map[39] = 0x4
map[40] = 0x4 map[41] = 0x4 map[42] = 0x4
map[43] = 0x4
map[44] = 0x4 map[45] = 0x4 map[46] = 0x4
map[47] = 0x4
map[48] = 0x4 map[49] = 0x4
TIMERS:
watchdog_timer : init_cnt = 0x0000A cnt = 0x00001 enable = 1
general_timer_0: init_cnt = 0x00320 cnt = 0x0031F cfg = 0
general_timer_1: init_cnt = 0xF4240 cnt = 0x0E3C7 cfg = 3
xxx#show eobc
Interface information:
Interface EOBC0/0 (idb = 0x47E585BC)
Hardware is Mistral EOBC (revision 5)
Address is 0000.1500.0000 (bia 0000.1500.0000)
Encap size = 14 hardware status = 0x210840
IDB type = 18 IDB state = 4
Encap type = 0x1 Span encap size = 0
Error threshold = 5000 Error count = 0
Counters:
rxring = 0x921D940 rx ring entries = 512
rx_head = 408 rx_tail = 0
inputs = 23432223331 rx_cumbytes =
8978218216407
hw inputs = 0 hw rx_cumbytes = 0
rx rate (bits/sec) = 613000 rx rate (packets/sec) = 156
rx_buf_unavail = 0 rx input drops = 56
input broadcast = 22 input resource = 2522086608
input error = 0 input giants = 0
input crc = 56 rx illegal length = 0
rxr eobc shadow = 0x47F65CB8 txr eobc shadow = 0x50ADB290
txring = 0x921F980 tx ring entries = 0x200
tx_head = 400 tx_tail = 400
outputs = 23315837396 tx_cumbytes =
1742852038156
hw outputs = 0 hw tx_cumbytes = 0
tx rate (bits/sec) = 93000 tx rate (packets/sec) = 156
tx_retry_error = 72 tx_retry_count = 6325477
tx_process_stopped = 17 tx total drops = 0
Mistral Registers
soft_reset_cfg = 0x040000 dma_buffer_size_reg = 0x000000
int_mask_hi = 0x00007E int_mask_lo = 0xE7001A58
rxdscp_cnt = 512 txdscp_cnt = 0
rxwork_dscp = 0xF2C0 txwork_dscp = 0x600
mistral_eobc_ds = 0x509CD908 mistral_dma_register = 0x30000000
mistral_glbl_reg = 0x10020000
Misc. Global Registers:
global_cfg = 0x20 mis_init_sts = 0xF
dimm_parm_cfg_hi = 0x00000576 dimm_parm_cfg_lo = 0x42040F5A
tm_init_size_cfg = 0x8000
xxx#
2018-03-01 10:02 GMT+01:00 James Bensley <jwbensley at gmail.com>:
> On 1 March 2018 at 08:29, james list <jameslist72 at gmail.com> wrote:
> > Dear experts,
> > has anybody experienced a 100% SP CPU usage on C6500-Sup720
> (12.2(33)SXI5)
> > with a lot of interrupts ?
> > The main process is Heartbeat.
> >
> > Cisco TAC is struggling in having an idea to sorting out the issue, they
> > are working since 3 days on it..
> >
> > STP is stable, no mac moving, no real issue found… maybe somebody
> > experienced the same due to something in particular?
> >
> > Thanks for any hints.
> >
> > Cheers,
> > James
> >
> >
> > xxx#remote command switch show process cpu sorted
> >
> > CPU utilization for five seconds: 91%/83%; one minute: 96%; five minutes:
> > 97%
>
> When you say a lot of interrupts, what do you get from:
>
> show platform netint
> remote command switch show platform hardware earl status
> show ibc
> show eobc
>
> I don't know what that Heartbeat process is for, e.g. between SP and
> RP, or SP and DFCs, or SP and line cards etc. In terms of fixing the
> issue, perhaps reboot the RSP or line card? That obviously doesn't
> give you a root cause though :)
>
> It seems like the process is stuck in a loop if you are saying that
> forwarding is working without issue.
>
> You could run a NetDR capture to see if that is control-plane traffic
> and maybe where its coming from or going to:
> https://null.53bits.co.uk/index.php?page=netdr-captures
>
> Cheers,
> James.
>
More information about the cisco-nsp
mailing list