[c-nsp] POS errors causing CPU spikes

Vincent De Keyzer vincent at dekeyzer.net
Thu Dec 22 05:25:58 EST 2005


Hello guys,

this is the second time I see errors on a POS controller (due to fibre
problems) that cause CPU spikes.

But when I look at what the CPU is doing exactly, I see nothing! :

BRUBLUro71#sh processes cpu sorted
CPU utilization for five seconds: 99%/98%; one minute: 99%; five minutes:
99%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  48    30984656 350984444         88  0.87%  0.80%  0.81%   0 IP Input
 155       10904       647      16853  0.39%  0.05%  0.01%   5 Virtual Exec
   4    11286180    591946      19066  0.23%  0.17%  0.21%   0 Check heaps
 142       17604    250045         70  0.07%  0.01%  0.00%   0 PDU
DISPATCHER
  89      412288   3154523        130  0.07%  0.16%  0.15%   0 CEF: IPv4
proces
   6           0         2          0  0.00%  0.00%  0.00%   0 Timers
   7           0         2          0  0.00%  0.00%  0.00%   0 Serial
Backgroun
   8         260    410367          0  0.00%  0.00%  0.00%   0
ALARM_TRIGGER_SC
   9           0         1          0  0.00%  0.00%  0.00%   0 Policy
Manager
  10      532440   2049241        259  0.00%  0.00%  0.00%   0 EnvMon
  11        1280         7     182857  0.00%  0.00%  0.00%   0 OIR Handler
   3       68780   1958263         35  0.00%  0.00%  0.00%   0 OSPF Hello
  13           0         1          0  0.00%  0.00%  0.00%   0 IPC Zone
Manager
   5          88       541        162  0.00%  0.00%  0.00%   0 Pool Manager
  15         920   2048163          0  0.00%  0.00%  0.00%   0 IPC Deferred
Por
  16           0         1          0  0.00%  0.00%  0.00%   0 IPC Seat
Manager
  17          96    205467          0  0.00%  0.00%  0.00%   0 Compute SRP
rate
   2        2104    410791          5  0.00%  0.00%  0.00%   0 Load Meter
  19           4        20        200  0.00%  0.00%  0.00%   0 DDR Timers
  12         252     34217          7  0.00%  0.00%  0.00%   0 IPC Dynamic
Cach
  21           0         4          0  0.00%  0.00%  0.00%   0 Entity MIB
API
  22           0         1          0  0.00%  0.00%  0.00%   0 SERIAL
A'detect
  14        1304   2048165          0  0.00%  0.00%  0.00%   0 IPC Periodic
Tim
  18     1933836  10875328        177  0.00%  0.10%  0.08%   0 ARP Input
  25     2560296    326626       7838  0.00%  0.00%  0.00%   0 Net
Background
  26        1568       885       1771  0.00%  0.00%  0.00%   0 Logger
  27        4248   2048154          2  0.00%  0.00%  0.00%   0 TTY
Background
  28       14348   2050629          6  0.00%  0.00%  0.00%   0 Per-Second
Jobs
  29      362764     35424      10240  0.00%  0.01%  0.00%   0 Per-minute
Jobs
  30           0         1          0  0.00%  0.00%  0.00%   0 Inode Table
Dest
  31           0         1          0  0.00%  0.00%  0.00%   0 CSP Timer
  32           0         2          0  0.00%  0.00%  0.00%   0 CES Timer
  33           0         2          0  0.00%  0.00%  0.00%   0 Hawkeye
Backgrou
  34           0         1          0  0.00%  0.00%  0.00%   0 SONET alarm
time
  35           0         2          0  0.00%  0.00%  0.00%   0 POS APS Event
Pr
  20           0         2          0  0.00%  0.00%  0.00%   0 Dialer event
  37           0         1          0  0.00%  0.00%  0.00%   0 CES Line
Conditi
  38           0         2          0  0.00%  0.00%  0.00%   0 Flash MIB
Update
  23        6460    446067         14  0.00%  0.00%  0.00%   0 HC Counter
Timer
  40           0         2          0  0.00%  0.00%  0.00%   0 ATM OAM TIMER
  41         316    412915          0  0.00%  0.00%  0.00%   0 ATM Periodic
  42           0         1          0  0.00%  0.00%  0.00%   0 ATM ARP INPUT
  43         932   2048738          0  0.00%  0.00%  0.00%   0 ATM PA Helper
  44           0         6          0  0.00%  0.00%  0.00%   0 TurboACL
  24           0         1          0  0.00%  0.00%  0.00%   0 Critical
Bkgnd
  46           0         1          0  0.00%  0.00%  0.00%   0 AC Switch
  47           0         2          0  0.00%  0.00%  0.00%   0 AAA
Dictionary R
  36           0         2          0  0.00%  0.00%  0.00%   0 VNM DSPRM
MAIN
  49           0         1          0  0.00%  0.00%  0.00%   0 ICMP event
handl
  50       29616    344412         85  0.00%  0.00%  0.00%   0 CDP Protocol
  51          76        43       1767  0.00%  0.00%  0.00%   0 PPP auth
  52          76        92        826  0.00%  0.00%  0.00%   0 OLM
  53           0         1          0  0.00%  0.00%  0.00%   0 PPPATM
Session d
  54           0         2          0  0.00%  0.00%  0.00%   0 PASVC create
VA
  55         292       513        569  0.00%  0.00%  0.00%   0 EEM ED Syslog
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  56           0         2          0  0.00%  0.00%  0.00%   0 EEM ED SNMP
  57           0         2          0  0.00%  0.00%  0.00%   0 EEM ED Memory
Th
  58          92     34352          2  0.00%  0.00%  0.00%   0 EEM ED Timer
  59           0         2          0  0.00%  0.00%  0.00%   0 EEM ED
Counter
  60           0         2          0  0.00%  0.00%  0.00%   0 EEM ED
Interface
  61           0         3          0  0.00%  0.00%  0.00%   0 EEM ED IOSWD
  62       91008   2143076         42  0.00%  0.01%  0.00%   0 HSRP
(Standby)
  63           0         1          0  0.00%  0.00%  0.00%   0 LSP Tunnel
FRR
  64           0         1          0  0.00%  0.00%  0.00%   0 MPLS
Auto-Tunnel
  65           0         1          0  0.00%  0.00%  0.00%   0 O-UNI Client
Msg
  66           0         2          0  0.00%  0.00%  0.00%   0 Spanning Tree
  67           0         1          0  0.00%  0.00%  0.00%   0 AC Mgr
  68           0         1          0  0.00%  0.00%  0.00%   0 SSS Manager
  69           0         2          0  0.00%  0.00%  0.00%   0 SSM
connection m
  70         164        34       4823  0.00%  0.00%  0.00%   0 PPP IP Add
Route
   1           0         2          0  0.00%  0.00%  0.00%   0 Chunk Manager
  72           0         2          0  0.00%  0.00%  0.00%   0 SSCOP Output
  73         348     34218         10  0.00%  0.00%  0.00%   0 SSCOP Timer
  74        1256     34215         36  0.00%  0.00%  0.00%   0 CEF
background
  75           0         1          0  0.00%  0.00%  0.00%   0 IP IRDP
  76           0         2          0  0.00%  0.00%  0.00%   0 SNMP Timers
  39           0         2          0  0.00%  0.00%  0.00%   0 ATM OAM Input
  45           0         2          0  0.00%  0.00%  0.00%   0 CEF switching
ba
  79           0         2          0  0.00%  0.00%  0.00%   0 ILMI Response
  80           0         1          0  0.00%  0.00%  0.00%   0 ILMI Timer
Proce
  81           4         2       2000  0.00%  0.00%  0.00%   0 ATM PVC
Discover
  82           0         2          0  0.00%  0.00%  0.00%   0 ATMSIG ILMI
Time
  83           0         2          0  0.00%  0.00%  0.00%   0 ATMSIG
DRIVERAPI
  84        1940   2048161          0  0.00%  0.00%  0.00%   0 ATMSIG Timer
  85           0         2          0  0.00%  0.00%  0.00%   0 ATMSIG Input
  86           0         2          0  0.00%  0.00%  0.00%   0 ATMSIG Client
  87     3131652     34522      90714  0.00%  0.14%  0.18%   0 IP Background
  88      232712    205587       1131  0.00%  0.00%  0.00%   0 IP RIB Update
  71           0         2          0  0.00%  0.00%  0.00%   0 SSCOP Input
  90         148       102       1450  0.00%  0.00%  0.00%   0 ADJ
background
  91           0         1          0  0.00%  0.00%  0.00%   0 L2X Data
Daemon
  92           0         1          0  0.00%  0.00%  0.00%   0 HTTP Timer
  93        4020    684289          5  0.00%  0.00%  0.00%   0 TCP Timer
  94         124       214        579  0.00%  0.00%  0.00%   0 TCP Protocols
  95           0         1          0  0.00%  0.00%  0.00%   0 Probe Input
  96           0        15          0  0.00%  0.00%  0.00%   0 RARP Input
  97           0         1          0  0.00%  0.00%  0.00%   0 Socket Timers
  98      286436   1013668        282  0.00%  0.00%  0.00%   0 DHCPD Receive
  99           0         1          0  0.00%  0.00%  0.00%   0 COPS
 100           0         1          0  0.00%  0.00%  0.00%   0 X.25 Encaps
Mana
 101           0         1          0  0.00%  0.00%  0.00%   0 PAD InCall
 102           0         2          0  0.00%  0.00%  0.00%   0 X.25
Background
 103           0         1          0  0.00%  0.00%  0.00%   0 AToM manager
 104           0         1          0  0.00%  0.00%  0.00%   0 AToM LDP
manager
 105           0         2          0  0.00%  0.00%  0.00%   0 LDP
Background
 106           0         1          0  0.00%  0.00%  0.00%   0 L2X Socket
proce
 107           0         1          0  0.00%  0.00%  0.00%   0 L2X SSS
manager
 108           0         1          0  0.00%  0.00%  0.00%   0 L2TP mgmt
daemon
 110           4         1       4000  0.00%  0.00%  0.00%   0 TSP
 111           0         1          0  0.00%  0.00%  0.00%   0 VOIP RTP Udp
Inp
 112           0         1          0  0.00%  0.00%  0.00%   0
QOS_MODULE_MAIN
 113           0         1          0  0.00%  0.00%  0.00%   0 CCVPM_HTSP
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 114           0         1          0  0.00%  0.00%  0.00%   0 CCVPM_R2
 115           0         1          0  0.00%  0.00%  0.00%   0 CCSWVOICE
 116         384    205215          1  0.00%  0.00%  0.00%   0 RMON Recycle
Pro
 117           0         2          0  0.00%  0.00%  0.00%   0 RMON Deferred
Se
 118           0         1          0  0.00%  0.00%  0.00%   0 SYSMGT Events
 119         120        56       2142  0.00%  0.00%  0.00%   0 SONET Traps
 120        1012    410433          2  0.00%  0.00%  0.00%   0 EEM Server
 121           0         1          0  0.00%  0.00%  0.00%   0 Syslog Traps
 122           0         1          0  0.00%  0.00%  0.00%   0 DATA Transfer
Pr
 123           0         1          0  0.00%  0.00%  0.00%   0 DATA
Collector
 124           0         1          0  0.00%  0.00%  0.00%   0 RMON Packets
 125           0         2          0  0.00%  0.00%  0.00%   0 EEM Policy
Direc
 126         964   2047199          0  0.00%  0.00%  0.00%   0 trunk
conditioni
 127           0         1          0  0.00%  0.00%  0.00%   0 trunk
conditioni
 128       87940   6339827         13  0.00%  0.00%  0.00%   0 Net Input
 129        7300    410703         17  0.00%  0.00%  0.00%   0 Compute load
avg
 130       11076   2211834          5  0.00%  0.00%  0.00%   0 NTP
 131           0         2          0  0.00%  0.00%  0.00%   0 CCP manager
 132           0         2          0  0.00%  0.00%  0.00%   0 CCP reset pak
 133        1868       662       2821  0.00%  0.00%  0.00%   0 PPP manager
 134         828   2048736          0  0.00%  0.00%  0.00%   0 Multilink PPP
 135           0         2          0  0.00%  0.00%  0.00%   0 Multilink PPP
ou
 136           0         2          0  0.00%  0.00%  0.00%   0 Multilink
event
 137         228       132       1727  0.00%  0.00%  0.00%   2 Virtual Exec
 138           0         1          0  0.00%  0.00%  0.00%   0 LSDp Input
Proc
 139         764    410433          1  0.00%  0.00%  0.00%   0 LSD Main Proc
 140           0        10          0  0.00%  0.00%  0.00%   0 TCP Listener
 141       50576    499240        101  0.00%  0.00%  0.00%   0 IP SNMP
  77           0         2          0  0.00%  0.00%  0.00%   0 ILMI Input
 143      129692    251067        516  0.00%  0.04%  0.04%   0 SNMP ENGINE
 144        3208        20     160400  0.00%  0.00%  0.00%   0 SNMP
ConfCopyPro
 145        2504        62      40387  0.00%  0.00%  0.00%   0 SNMP Traps
 146           0         1          0  0.00%  0.00%  0.00%   0 xcpa-driver
 147          28     17128          1  0.00%  0.00%  0.00%   0 DHCPD Timer
 148        1600    581436          2  0.00%  0.00%  0.00%   0 DHCPD
Database
 149       53468   2380789         22  0.00%  0.00%  0.00%   0 OSPF Router
1755
 150     5432868   5481079        991  0.00%  0.30%  0.39%   0 BGP Router
 151      117732    797564        147  0.00%  0.00%  0.00%   0 BGP I/O
 152    57771960    387731     149001  0.00%  0.99%  1.45%   0 BGP Scanner
 153        7412       247      30008  0.00%  0.00%  0.00%   3 Virtual Exec
 154        8720       200      43600  0.00%  0.00%  0.04%   4 Virtual Exec
  78           0         2          0  0.00%  0.00%  0.00%   0 ILMI Request
BRUBLUro71#

How come the numbers do not add up?

Vincent



More information about the cisco-nsp mailing list