[c-nsp] POS errors causing CPU spikes
Vincent De Keyzer
vincent at dekeyzer.net
Thu Dec 22 11:06:24 EST 2005
Olivier,
thanks for your reply.
It's 7206VXR w/ NPE-G1 (but I think I have seen it on NPE-400 too). Traffic
isn't particularly high. Problem disappeared when we reconnected the fibre
correctly. Can't do a "show align" hence.
... can't afford TAC support either :)
Vincent
> -----Original Message-----
> From: Oliver Boehmer (oboehmer) [mailto:oboehmer at cisco.com]
> Sent: jeudi 22 décembre 2005 12:13
> To: Vincent De Keyzer; cisco-nsp at puck.nether.net
> Subject: RE: [c-nsp] POS errors causing CPU spikes
>
> Vincent,
>
> your CPU spends all its time in interrupt (the 2nd figure in "99%/98%"),
> hence you don't see any high util in any of the processes. Which
> platform is this? Do you see high traffic rate anywhere else? Do you see
> anythin in "show align"?
> best open a TAC case, things like this can require some deeper digging..
>
> oli
>
> Vincent De Keyzer <> wrote on Thursday, December 22, 2005 11:26 AM:
>
> > Hello guys,
> >
> > this is the second time I see errors on a POS controller (due to fibre
> > problems) that cause CPU spikes.
> >
> > But when I look at what the CPU is doing exactly, I see nothing! :
> >
> > BRUBLUro71#sh processes cpu sorted
> > CPU utilization for five seconds: 99%/98%; one minute: 99%; five
> > minutes: 99%
> > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> > 48 30984656 350984444 88 0.87% 0.80% 0.81% 0 IP
> > Input 155 10904 647 16853 0.39% 0.05% 0.01% 5
> > Virtual Exec 4 11286180 591946 19066 0.23% 0.17%
> > 0.21% 0 Check heaps 142 17604 250045 70 0.07%
> > 0.01% 0.00% 0 PDU DISPATCHER
> > 89 412288 3154523 130 0.07% 0.16% 0.15% 0 CEF:
> > IPv4 proces
> > 6 0 2 0 0.00% 0.00% 0.00% 0 Timers
> > 7 0 2 0 0.00% 0.00% 0.00% 0 Serial
> > Backgroun
> > 8 260 410367 0 0.00% 0.00% 0.00% 0
> > ALARM_TRIGGER_SC
> > 9 0 1 0 0.00% 0.00% 0.00% 0 Policy
> > Manager
> > 10 532440 2049241 259 0.00% 0.00% 0.00% 0 EnvMon
> > 11 1280 7 182857 0.00% 0.00% 0.00% 0 OIR
> > Handler 3 68780 1958263 35 0.00% 0.00% 0.00%
> > 0 OSPF Hello 13 0 1 0 0.00% 0.00%
> > 0.00% 0 IPC Zone Manager
> > 5 88 541 162 0.00% 0.00% 0.00% 0 Pool
> > Manager 15 920 2048163 0 0.00% 0.00% 0.00%
> > 0 IPC Deferred Por
> > 16 0 1 0 0.00% 0.00% 0.00% 0 IPC
> > Seat Manager
> > 17 96 205467 0 0.00% 0.00% 0.00% 0
> > Compute SRP rate
> > 2 2104 410791 5 0.00% 0.00% 0.00% 0 Load
> > Meter 19 4 20 200 0.00% 0.00% 0.00% 0
> > DDR Timers 12 252 34217 7 0.00% 0.00% 0.00%
> > 0 IPC Dynamic Cach
> > 21 0 4 0 0.00% 0.00% 0.00% 0 Entity
> > MIB API
> > 22 0 1 0 0.00% 0.00% 0.00% 0 SERIAL
> > A'detect
> > 14 1304 2048165 0 0.00% 0.00% 0.00% 0 IPC
> > Periodic Tim
> > 18 1933836 10875328 177 0.00% 0.10% 0.08% 0 ARP
> > Input 25 2560296 326626 7838 0.00% 0.00% 0.00% 0
> > Net
> > Background
> > 26 1568 885 1771 0.00% 0.00% 0.00% 0 Logger
> > 27 4248 2048154 2 0.00% 0.00% 0.00% 0 TTY
> > Background
> > 28 14348 2050629 6 0.00% 0.00% 0.00% 0
> > Per-Second Jobs
> > 29 362764 35424 10240 0.00% 0.01% 0.00% 0
> > Per-minute Jobs
> > 30 0 1 0 0.00% 0.00% 0.00% 0 Inode
> > Table Dest
> > 31 0 1 0 0.00% 0.00% 0.00% 0 CSP
> > Timer 32 0 2 0 0.00% 0.00% 0.00% 0
> > CES Timer 33 0 2 0 0.00% 0.00% 0.00%
> > 0 Hawkeye Backgrou
> > 34 0 1 0 0.00% 0.00% 0.00% 0 SONET
> > alarm time
> > 35 0 2 0 0.00% 0.00% 0.00% 0 POS
> > APS Event Pr
> > 20 0 2 0 0.00% 0.00% 0.00% 0 Dialer
> > event 37 0 1 0 0.00% 0.00% 0.00% 0
> > CES Line Conditi
> > 38 0 2 0 0.00% 0.00% 0.00% 0 Flash
> > MIB Update
> > 23 6460 446067 14 0.00% 0.00% 0.00% 0 HC
> > Counter Timer
> > 40 0 2 0 0.00% 0.00% 0.00% 0 ATM
> > OAM TIMER 41 316 412915 0 0.00% 0.00% 0.00%
> > 0 ATM Periodic 42 0 1 0 0.00% 0.00%
> > 0.00% 0 ATM ARP INPUT 43 932 2048738 0 0.00%
> > 0.00% 0.00% 0 ATM PA Helper 44 0 6 0
> > 0.00% 0.00% 0.00% 0 TurboACL 24 0 1
> > 0 0.00% 0.00% 0.00% 0 Critical
> > Bkgnd
> > 46 0 1 0 0.00% 0.00% 0.00% 0 AC
> > Switch 47 0 2 0 0.00% 0.00% 0.00% 0
> > AAA
> > Dictionary R
> > 36 0 2 0 0.00% 0.00% 0.00% 0 VNM
> > DSPRM MAIN
> > 49 0 1 0 0.00% 0.00% 0.00% 0 ICMP
> > event handl
> > 50 29616 344412 85 0.00% 0.00% 0.00% 0 CDP
> > Protocol 51 76 43 1767 0.00% 0.00% 0.00%
> > 0 PPP auth 52 76 92 826 0.00% 0.00% 0.00%
> > 0 OLM 53 0 1 0 0.00% 0.00% 0.00% 0
> > PPPATM
> > Session d
> > 54 0 2 0 0.00% 0.00% 0.00% 0 PASVC
> > create VA
> > 55 292 513 569 0.00% 0.00% 0.00% 0 EEM ED
> > Syslog PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY
> > Process 56 0 2 0 0.00% 0.00% 0.00%
> > 0 EEM ED SNMP 57 0 2 0 0.00% 0.00%
> > 0.00% 0 EEM ED Memory Th
> > 58 92 34352 2 0.00% 0.00% 0.00% 0 EEM ED
> > Timer 59 0 2 0 0.00% 0.00% 0.00% 0
> > EEM ED
> > Counter
> > 60 0 2 0 0.00% 0.00% 0.00% 0 EEM ED
> > Interface
> > 61 0 3 0 0.00% 0.00% 0.00% 0 EEM ED
> > IOSWD 62 91008 2143076 42 0.00% 0.01% 0.00% 0
> > HSRP (Standby)
> > 63 0 1 0 0.00% 0.00% 0.00% 0 LSP
> > Tunnel FRR
> > 64 0 1 0 0.00% 0.00% 0.00% 0 MPLS
> > Auto-Tunnel
> > 65 0 1 0 0.00% 0.00% 0.00% 0 O-UNI
> > Client Msg
> > 66 0 2 0 0.00% 0.00% 0.00% 0
> > Spanning Tree 67 0 1 0 0.00% 0.00%
> > 0.00% 0 AC Mgr 68 0 1 0 0.00% 0.00%
> > 0.00% 0 SSS Manager 69 0 2 0 0.00%
> > 0.00% 0.00% 0 SSM
> > connection m
> > 70 164 34 4823 0.00% 0.00% 0.00% 0 PPP IP
> > Add Route
> > 1 0 2 0 0.00% 0.00% 0.00% 0 Chunk
> > Manager 72 0 2 0 0.00% 0.00% 0.00%
> > 0 SSCOP Output 73 348 34218 10 0.00% 0.00%
> > 0.00% 0 SSCOP Timer 74 1256 34215 36 0.00%
> > 0.00% 0.00% 0 CEF
> > background
> > 75 0 1 0 0.00% 0.00% 0.00% 0 IP IRDP
> > 76 0 2 0 0.00% 0.00% 0.00% 0 SNMP
> > Timers 39 0 2 0 0.00% 0.00% 0.00% 0
> > ATM OAM Input 45 0 2 0 0.00% 0.00%
> > 0.00% 0 CEF switching ba
> > 79 0 2 0 0.00% 0.00% 0.00% 0 ILMI
> > Response 80 0 1 0 0.00% 0.00% 0.00%
> > 0 ILMI Timer Proce
> > 81 4 2 2000 0.00% 0.00% 0.00% 0 ATM PVC
> > Discover
> > 82 0 2 0 0.00% 0.00% 0.00% 0 ATMSIG
> > ILMI Time
> > 83 0 2 0 0.00% 0.00% 0.00% 0 ATMSIG
> > DRIVERAPI
> > 84 1940 2048161 0 0.00% 0.00% 0.00% 0 ATMSIG
> > Timer 85 0 2 0 0.00% 0.00% 0.00% 0
> > ATMSIG Input 86 0 2 0 0.00% 0.00%
> > 0.00% 0 ATMSIG Client 87 3131652 34522 90714 0.00%
> > 0.14% 0.18% 0 IP Background 88 232712 205587 1131
> > 0.00% 0.00% 0.00% 0 IP RIB Update 71 0 2
> > 0 0.00% 0.00% 0.00% 0 SSCOP Input 90 148 102
> > 1450 0.00% 0.00% 0.00% 0 ADJ
> > background
> > 91 0 1 0 0.00% 0.00% 0.00% 0 L2X
> > Data Daemon
> > 92 0 1 0 0.00% 0.00% 0.00% 0 HTTP
> > Timer 93 4020 684289 5 0.00% 0.00% 0.00% 0
> > TCP Timer 94 124 214 579 0.00% 0.00% 0.00%
> > 0 TCP Protocols 95 0 1 0 0.00% 0.00%
> > 0.00% 0 Probe Input 96 0 15 0 0.00%
> > 0.00% 0.00% 0 RARP Input 97 0 1 0
> > 0.00% 0.00% 0.00% 0 Socket Timers 98 286436 1013668
> > 282 0.00% 0.00% 0.00% 0 DHCPD Receive 99 0 1
> > 0 0.00% 0.00% 0.00% 0 COPS 100 0 1 0
> > 0.00% 0.00% 0.00% 0 X.25 Encaps Mana
> > 101 0 1 0 0.00% 0.00% 0.00% 0 PAD
> > InCall 102 0 2 0 0.00% 0.00% 0.00% 0
> > X.25 Background
> > 103 0 1 0 0.00% 0.00% 0.00% 0 AToM
> > manager 104 0 1 0 0.00% 0.00% 0.00%
> > 0 AToM LDP manager
> > 105 0 2 0 0.00% 0.00% 0.00% 0 LDP
> > Background
> > 106 0 1 0 0.00% 0.00% 0.00% 0 L2X
> > Socket proce
> > 107 0 1 0 0.00% 0.00% 0.00% 0 L2X SSS
> > manager
> > 108 0 1 0 0.00% 0.00% 0.00% 0 L2TP
> > mgmt daemon
> > 110 4 1 4000 0.00% 0.00% 0.00% 0 TSP
> > 111 0 1 0 0.00% 0.00% 0.00% 0 VOIP
> > RTP Udp Inp
> > 112 0 1 0 0.00% 0.00% 0.00% 0
> > QOS_MODULE_MAIN
> > 113 0 1 0 0.00% 0.00% 0.00% 0
> > CCVPM_HTSP PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min
> > TTY Process 114 0 1 0 0.00% 0.00%
> > 0.00% 0 CCVPM_R2 115 0 1 0 0.00%
> > 0.00% 0.00% 0 CCSWVOICE 116 384 205215 1
> > 0.00% 0.00% 0.00% 0 RMON Recycle Pro
> > 117 0 2 0 0.00% 0.00% 0.00% 0 RMON
> > Deferred Se
> > 118 0 1 0 0.00% 0.00% 0.00% 0 SYSMGT
> > Events 119 120 56 2142 0.00% 0.00% 0.00% 0
> > SONET Traps 120 1012 410433 2 0.00% 0.00%
> > 0.00% 0 EEM Server 121 0 1 0 0.00%
> > 0.00% 0.00% 0 Syslog Traps 122 0 1 0
> > 0.00% 0.00% 0.00% 0 DATA Transfer Pr
> > 123 0 1 0 0.00% 0.00% 0.00% 0 DATA
> > Collector
> > 124 0 1 0 0.00% 0.00% 0.00% 0 RMON
> > Packets 125 0 2 0 0.00% 0.00% 0.00%
> > 0 EEM Policy Direc
> > 126 964 2047199 0 0.00% 0.00% 0.00% 0 trunk
> > conditioni
> > 127 0 1 0 0.00% 0.00% 0.00% 0 trunk
> > conditioni
> > 128 87940 6339827 13 0.00% 0.00% 0.00% 0 Net
> > Input 129 7300 410703 17 0.00% 0.00% 0.00% 0
> > Compute load avg
> > 130 11076 2211834 5 0.00% 0.00% 0.00% 0 NTP
> > 131 0 2 0 0.00% 0.00% 0.00% 0 CCP
> > manager 132 0 2 0 0.00% 0.00% 0.00%
> > 0 CCP reset pak 133 1868 662 2821 0.00% 0.00%
> > 0.00% 0 PPP manager 134 828 2048736 0 0.00%
> > 0.00% 0.00% 0 Multilink PPP 135 0 2 0
> > 0.00% 0.00% 0.00% 0 Multilink PPP ou
> > 136 0 2 0 0.00% 0.00% 0.00% 0
> > Multilink event
> > 137 228 132 1727 0.00% 0.00% 0.00% 2
> > Virtual Exec 138 0 1 0 0.00% 0.00%
> > 0.00% 0 LSDp Input Proc
> > 139 764 410433 1 0.00% 0.00% 0.00% 0 LSD
> > Main Proc 140 0 10 0 0.00% 0.00% 0.00%
> > 0 TCP Listener 141 50576 499240 101 0.00% 0.00%
> > 0.00% 0 IP SNMP 77 0 2 0 0.00% 0.00%
> > 0.00% 0 ILMI Input 143 129692 251067 516 0.00%
> > 0.04% 0.04% 0 SNMP ENGINE 144 3208 20 160400
> > 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
> > 145 2504 62 40387 0.00% 0.00% 0.00% 0 SNMP
> > Traps 146 0 1 0 0.00% 0.00% 0.00% 0
> > xcpa-driver 147 28 17128 1 0.00% 0.00%
> > 0.00% 0 DHCPD Timer 148 1600 581436 2 0.00%
> > 0.00% 0.00% 0 DHCPD Database
> > 149 53468 2380789 22 0.00% 0.00% 0.00% 0 OSPF
> > Router 1755
> > 150 5432868 5481079 991 0.00% 0.30% 0.39% 0 BGP
> > Router 151 117732 797564 147 0.00% 0.00% 0.00% 0
> > BGP I/O 152 57771960 387731 149001 0.00% 0.99% 1.45%
> > 0 BGP Scanner 153 7412 247 30008 0.00% 0.00%
> > 0.00% 3 Virtual Exec 154 8720 200 43600 0.00%
> > 0.00% 0.04% 4 Virtual Exec 78 0 2 0
> > 0.00% 0.00% 0.00% 0 ILMI Request BRUBLUro71#
> >
> > How come the numbers do not add up?
> >
> > Vincent
> >
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list