[c-nsp] POS errors causing CPU spikes
Oliver Boehmer (oboehmer)
oboehmer at cisco.com
Thu Dec 22 06:12:50 EST 2005
Vincent,
your CPU spends all its time in interrupt (the 2nd figure in "99%/98%"),
hence you don't see any high util in any of the processes. Which
platform is this? Do you see high traffic rate anywhere else? Do you see
anythin in "show align"?
best open a TAC case, things like this can require some deeper digging..
oli
Vincent De Keyzer <> wrote on Thursday, December 22, 2005 11:26 AM:
> Hello guys,
>
> this is the second time I see errors on a POS controller (due to fibre
> problems) that cause CPU spikes.
>
> But when I look at what the CPU is doing exactly, I see nothing! :
>
> BRUBLUro71#sh processes cpu sorted
> CPU utilization for five seconds: 99%/98%; one minute: 99%; five
> minutes: 99%
> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> 48 30984656 350984444 88 0.87% 0.80% 0.81% 0 IP
> Input 155 10904 647 16853 0.39% 0.05% 0.01% 5
> Virtual Exec 4 11286180 591946 19066 0.23% 0.17%
> 0.21% 0 Check heaps 142 17604 250045 70 0.07%
> 0.01% 0.00% 0 PDU DISPATCHER
> 89 412288 3154523 130 0.07% 0.16% 0.15% 0 CEF:
> IPv4 proces
> 6 0 2 0 0.00% 0.00% 0.00% 0 Timers
> 7 0 2 0 0.00% 0.00% 0.00% 0 Serial
> Backgroun
> 8 260 410367 0 0.00% 0.00% 0.00% 0
> ALARM_TRIGGER_SC
> 9 0 1 0 0.00% 0.00% 0.00% 0 Policy
> Manager
> 10 532440 2049241 259 0.00% 0.00% 0.00% 0 EnvMon
> 11 1280 7 182857 0.00% 0.00% 0.00% 0 OIR
> Handler 3 68780 1958263 35 0.00% 0.00% 0.00%
> 0 OSPF Hello 13 0 1 0 0.00% 0.00%
> 0.00% 0 IPC Zone Manager
> 5 88 541 162 0.00% 0.00% 0.00% 0 Pool
> Manager 15 920 2048163 0 0.00% 0.00% 0.00%
> 0 IPC Deferred Por
> 16 0 1 0 0.00% 0.00% 0.00% 0 IPC
> Seat Manager
> 17 96 205467 0 0.00% 0.00% 0.00% 0
> Compute SRP rate
> 2 2104 410791 5 0.00% 0.00% 0.00% 0 Load
> Meter 19 4 20 200 0.00% 0.00% 0.00% 0
> DDR Timers 12 252 34217 7 0.00% 0.00% 0.00%
> 0 IPC Dynamic Cach
> 21 0 4 0 0.00% 0.00% 0.00% 0 Entity
> MIB API
> 22 0 1 0 0.00% 0.00% 0.00% 0 SERIAL
> A'detect
> 14 1304 2048165 0 0.00% 0.00% 0.00% 0 IPC
> Periodic Tim
> 18 1933836 10875328 177 0.00% 0.10% 0.08% 0 ARP
> Input 25 2560296 326626 7838 0.00% 0.00% 0.00% 0
> Net
> Background
> 26 1568 885 1771 0.00% 0.00% 0.00% 0 Logger
> 27 4248 2048154 2 0.00% 0.00% 0.00% 0 TTY
> Background
> 28 14348 2050629 6 0.00% 0.00% 0.00% 0
> Per-Second Jobs
> 29 362764 35424 10240 0.00% 0.01% 0.00% 0
> Per-minute Jobs
> 30 0 1 0 0.00% 0.00% 0.00% 0 Inode
> Table Dest
> 31 0 1 0 0.00% 0.00% 0.00% 0 CSP
> Timer 32 0 2 0 0.00% 0.00% 0.00% 0
> CES Timer 33 0 2 0 0.00% 0.00% 0.00%
> 0 Hawkeye Backgrou
> 34 0 1 0 0.00% 0.00% 0.00% 0 SONET
> alarm time
> 35 0 2 0 0.00% 0.00% 0.00% 0 POS
> APS Event Pr
> 20 0 2 0 0.00% 0.00% 0.00% 0 Dialer
> event 37 0 1 0 0.00% 0.00% 0.00% 0
> CES Line Conditi
> 38 0 2 0 0.00% 0.00% 0.00% 0 Flash
> MIB Update
> 23 6460 446067 14 0.00% 0.00% 0.00% 0 HC
> Counter Timer
> 40 0 2 0 0.00% 0.00% 0.00% 0 ATM
> OAM TIMER 41 316 412915 0 0.00% 0.00% 0.00%
> 0 ATM Periodic 42 0 1 0 0.00% 0.00%
> 0.00% 0 ATM ARP INPUT 43 932 2048738 0 0.00%
> 0.00% 0.00% 0 ATM PA Helper 44 0 6 0
> 0.00% 0.00% 0.00% 0 TurboACL 24 0 1
> 0 0.00% 0.00% 0.00% 0 Critical
> Bkgnd
> 46 0 1 0 0.00% 0.00% 0.00% 0 AC
> Switch 47 0 2 0 0.00% 0.00% 0.00% 0
> AAA
> Dictionary R
> 36 0 2 0 0.00% 0.00% 0.00% 0 VNM
> DSPRM MAIN
> 49 0 1 0 0.00% 0.00% 0.00% 0 ICMP
> event handl
> 50 29616 344412 85 0.00% 0.00% 0.00% 0 CDP
> Protocol 51 76 43 1767 0.00% 0.00% 0.00%
> 0 PPP auth 52 76 92 826 0.00% 0.00% 0.00%
> 0 OLM 53 0 1 0 0.00% 0.00% 0.00% 0
> PPPATM
> Session d
> 54 0 2 0 0.00% 0.00% 0.00% 0 PASVC
> create VA
> 55 292 513 569 0.00% 0.00% 0.00% 0 EEM ED
> Syslog PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY
> Process 56 0 2 0 0.00% 0.00% 0.00%
> 0 EEM ED SNMP 57 0 2 0 0.00% 0.00%
> 0.00% 0 EEM ED Memory Th
> 58 92 34352 2 0.00% 0.00% 0.00% 0 EEM ED
> Timer 59 0 2 0 0.00% 0.00% 0.00% 0
> EEM ED
> Counter
> 60 0 2 0 0.00% 0.00% 0.00% 0 EEM ED
> Interface
> 61 0 3 0 0.00% 0.00% 0.00% 0 EEM ED
> IOSWD 62 91008 2143076 42 0.00% 0.01% 0.00% 0
> HSRP (Standby)
> 63 0 1 0 0.00% 0.00% 0.00% 0 LSP
> Tunnel FRR
> 64 0 1 0 0.00% 0.00% 0.00% 0 MPLS
> Auto-Tunnel
> 65 0 1 0 0.00% 0.00% 0.00% 0 O-UNI
> Client Msg
> 66 0 2 0 0.00% 0.00% 0.00% 0
> Spanning Tree 67 0 1 0 0.00% 0.00%
> 0.00% 0 AC Mgr 68 0 1 0 0.00% 0.00%
> 0.00% 0 SSS Manager 69 0 2 0 0.00%
> 0.00% 0.00% 0 SSM
> connection m
> 70 164 34 4823 0.00% 0.00% 0.00% 0 PPP IP
> Add Route
> 1 0 2 0 0.00% 0.00% 0.00% 0 Chunk
> Manager 72 0 2 0 0.00% 0.00% 0.00%
> 0 SSCOP Output 73 348 34218 10 0.00% 0.00%
> 0.00% 0 SSCOP Timer 74 1256 34215 36 0.00%
> 0.00% 0.00% 0 CEF
> background
> 75 0 1 0 0.00% 0.00% 0.00% 0 IP IRDP
> 76 0 2 0 0.00% 0.00% 0.00% 0 SNMP
> Timers 39 0 2 0 0.00% 0.00% 0.00% 0
> ATM OAM Input 45 0 2 0 0.00% 0.00%
> 0.00% 0 CEF switching ba
> 79 0 2 0 0.00% 0.00% 0.00% 0 ILMI
> Response 80 0 1 0 0.00% 0.00% 0.00%
> 0 ILMI Timer Proce
> 81 4 2 2000 0.00% 0.00% 0.00% 0 ATM PVC
> Discover
> 82 0 2 0 0.00% 0.00% 0.00% 0 ATMSIG
> ILMI Time
> 83 0 2 0 0.00% 0.00% 0.00% 0 ATMSIG
> DRIVERAPI
> 84 1940 2048161 0 0.00% 0.00% 0.00% 0 ATMSIG
> Timer 85 0 2 0 0.00% 0.00% 0.00% 0
> ATMSIG Input 86 0 2 0 0.00% 0.00%
> 0.00% 0 ATMSIG Client 87 3131652 34522 90714 0.00%
> 0.14% 0.18% 0 IP Background 88 232712 205587 1131
> 0.00% 0.00% 0.00% 0 IP RIB Update 71 0 2
> 0 0.00% 0.00% 0.00% 0 SSCOP Input 90 148 102
> 1450 0.00% 0.00% 0.00% 0 ADJ
> background
> 91 0 1 0 0.00% 0.00% 0.00% 0 L2X
> Data Daemon
> 92 0 1 0 0.00% 0.00% 0.00% 0 HTTP
> Timer 93 4020 684289 5 0.00% 0.00% 0.00% 0
> TCP Timer 94 124 214 579 0.00% 0.00% 0.00%
> 0 TCP Protocols 95 0 1 0 0.00% 0.00%
> 0.00% 0 Probe Input 96 0 15 0 0.00%
> 0.00% 0.00% 0 RARP Input 97 0 1 0
> 0.00% 0.00% 0.00% 0 Socket Timers 98 286436 1013668
> 282 0.00% 0.00% 0.00% 0 DHCPD Receive 99 0 1
> 0 0.00% 0.00% 0.00% 0 COPS 100 0 1 0
> 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
> 101 0 1 0 0.00% 0.00% 0.00% 0 PAD
> InCall 102 0 2 0 0.00% 0.00% 0.00% 0
> X.25 Background
> 103 0 1 0 0.00% 0.00% 0.00% 0 AToM
> manager 104 0 1 0 0.00% 0.00% 0.00%
> 0 AToM LDP manager
> 105 0 2 0 0.00% 0.00% 0.00% 0 LDP
> Background
> 106 0 1 0 0.00% 0.00% 0.00% 0 L2X
> Socket proce
> 107 0 1 0 0.00% 0.00% 0.00% 0 L2X SSS
> manager
> 108 0 1 0 0.00% 0.00% 0.00% 0 L2TP
> mgmt daemon
> 110 4 1 4000 0.00% 0.00% 0.00% 0 TSP
> 111 0 1 0 0.00% 0.00% 0.00% 0 VOIP
> RTP Udp Inp
> 112 0 1 0 0.00% 0.00% 0.00% 0
> QOS_MODULE_MAIN
> 113 0 1 0 0.00% 0.00% 0.00% 0
> CCVPM_HTSP PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min
> TTY Process 114 0 1 0 0.00% 0.00%
> 0.00% 0 CCVPM_R2 115 0 1 0 0.00%
> 0.00% 0.00% 0 CCSWVOICE 116 384 205215 1
> 0.00% 0.00% 0.00% 0 RMON Recycle Pro
> 117 0 2 0 0.00% 0.00% 0.00% 0 RMON
> Deferred Se
> 118 0 1 0 0.00% 0.00% 0.00% 0 SYSMGT
> Events 119 120 56 2142 0.00% 0.00% 0.00% 0
> SONET Traps 120 1012 410433 2 0.00% 0.00%
> 0.00% 0 EEM Server 121 0 1 0 0.00%
> 0.00% 0.00% 0 Syslog Traps 122 0 1 0
> 0.00% 0.00% 0.00% 0 DATA Transfer Pr
> 123 0 1 0 0.00% 0.00% 0.00% 0 DATA
> Collector
> 124 0 1 0 0.00% 0.00% 0.00% 0 RMON
> Packets 125 0 2 0 0.00% 0.00% 0.00%
> 0 EEM Policy Direc
> 126 964 2047199 0 0.00% 0.00% 0.00% 0 trunk
> conditioni
> 127 0 1 0 0.00% 0.00% 0.00% 0 trunk
> conditioni
> 128 87940 6339827 13 0.00% 0.00% 0.00% 0 Net
> Input 129 7300 410703 17 0.00% 0.00% 0.00% 0
> Compute load avg
> 130 11076 2211834 5 0.00% 0.00% 0.00% 0 NTP
> 131 0 2 0 0.00% 0.00% 0.00% 0 CCP
> manager 132 0 2 0 0.00% 0.00% 0.00%
> 0 CCP reset pak 133 1868 662 2821 0.00% 0.00%
> 0.00% 0 PPP manager 134 828 2048736 0 0.00%
> 0.00% 0.00% 0 Multilink PPP 135 0 2 0
> 0.00% 0.00% 0.00% 0 Multilink PPP ou
> 136 0 2 0 0.00% 0.00% 0.00% 0
> Multilink event
> 137 228 132 1727 0.00% 0.00% 0.00% 2
> Virtual Exec 138 0 1 0 0.00% 0.00%
> 0.00% 0 LSDp Input Proc
> 139 764 410433 1 0.00% 0.00% 0.00% 0 LSD
> Main Proc 140 0 10 0 0.00% 0.00% 0.00%
> 0 TCP Listener 141 50576 499240 101 0.00% 0.00%
> 0.00% 0 IP SNMP 77 0 2 0 0.00% 0.00%
> 0.00% 0 ILMI Input 143 129692 251067 516 0.00%
> 0.04% 0.04% 0 SNMP ENGINE 144 3208 20 160400
> 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
> 145 2504 62 40387 0.00% 0.00% 0.00% 0 SNMP
> Traps 146 0 1 0 0.00% 0.00% 0.00% 0
> xcpa-driver 147 28 17128 1 0.00% 0.00%
> 0.00% 0 DHCPD Timer 148 1600 581436 2 0.00%
> 0.00% 0.00% 0 DHCPD Database
> 149 53468 2380789 22 0.00% 0.00% 0.00% 0 OSPF
> Router 1755
> 150 5432868 5481079 991 0.00% 0.30% 0.39% 0 BGP
> Router 151 117732 797564 147 0.00% 0.00% 0.00% 0
> BGP I/O 152 57771960 387731 149001 0.00% 0.99% 1.45%
> 0 BGP Scanner 153 7412 247 30008 0.00% 0.00%
> 0.00% 3 Virtual Exec 154 8720 200 43600 0.00%
> 0.00% 0.04% 4 Virtual Exec 78 0 2 0
> 0.00% 0.00% 0.00% 0 ILMI Request BRUBLUro71#
>
> How come the numbers do not add up?
>
> Vincent
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list