[c-nsp] High CPU caused by interupt on 7600 router
Rin
rintrum at gmail.com
Fri Oct 1 05:34:08 EDT 2010
Hi group,
I have a 7609 PE router with 2 TenGigabit interfaces uplink to other P
routers and Gigabit interfaces downlink to access switch. Recently, I detect
the CPU of that router is punted approximately every 35mins and result in
below output:
==============================================================
PE-Router#sho proc cpu hist
6666666644444555552222211111333331111133333111115555511111
100
90
80
70
60
50
40
30
20
10 ******** ***** *****
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
88 6 99
5664764335455448544554465856444344555656540444344454357766
100
90 ** **
80 ** **
70 ** * **
60 ** * **
50 ** * **
40 #* * **
30 #* * #*
20 #* * #*
10 *## #* * ** ** ** ***** *******## * *****
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
989968999998999999999979999988898989999887299889 8888 9999988989884889
7653706465425842251504506532186452964008644208308676970312489352156780
100 * * * ** ** * * ** *
90 **** ***** ********** ***** ************ *** * **** ********** * ***
80 **** ************************************ ***** **** ************ ***
70 ****************************************** ***** **** ************ ***
60 ****************************************** ***** **** ************ ***
50 ****************************************** ***** **** ****************
40 ****************************************** ***** **** ****************
30 ****************************************** ***** **** ****************
20 ************************************************ **** ****************
10 **********************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
==============================================================
I can confirm this caused by interrupt by configure ERM feature to detect
the interrupt on the router and check CPU right after the interrupt happens.
The following output display the logging:
============================================================================
===
*Sep 29 13:39:38.830: %SYS-4-CPURESRISING: System is seeing global cpu util
97% at total level more than the configured critical limit 80 %
*Sep 29 13:39:38.830: %SYS-4-CPURESRISING: System is seeing global cpu util
97% at interrupt level more than the configured critical limit 70 %
*Sep 29 13:39:48.830: %SYS-6-CPURESFALLING: System is no longer seeing
global high cpu at total level for the configured critical limit 80%,
current value 44%
*Sep 29 13:39:48.830: %SYS-6-CPURESFALLING: System is no longer seeing
global high cpu at interrupt level for the configured critical limit 70%,
current value 44%
*Sep 29 13:42:13.750: %SYS-5-CONFIG_I: Configured from console by admin on
vty0 (172.16.252.251)
*Sep 29 14:14:11.122: %SYS-5-CONFIG_I: Configured from console by admin on
vty0 (172.16.252.251)
*Sep 29 14:16:58.862: %SYS-4-CPURESRISING: System is seeing global cpu util
90% at total level more than the configured critical limit 80 %
*Sep 29 14:16:58.862: %SYS-4-CPURESRISING: System is seeing global cpu util
90% at interrupt level more than the configured critical limit 70 %
*Sep 29 14:17:08.862: %SYS-6-CPURESFALLING: System is no longer seeing
global high cpu at total level for the configured critical limit 80%,
current value 70%
*Sep 29 14:17:13.862: %SYS-6-CPURESFALLING: System is no longer seeing
global high cpu at interrupt level for the configured critical limit 70%,
current value 37%
*Sep 29 14:22:18.046: %SYS-5-CONFIG_I: Configured from console by admin on
vty0 (172.16.252.251)
*Sep 29 14:56:58.886: %SYS-4-CPURESRISING: System is seeing global cpu util
86% at total level more than the configured critical limit 80 %
*Sep 29 14:56:58.886: %SYS-4-CPURESRISING: System is seeing global cpu util
86% at interrupt level more than the configured critical limit 70 %
*Sep 29 14:57:08.886: %SYS-6-CPURESFALLING: System is no longer seeing
global high cpu at total level for the configured critical limit 80%,
current value 78%
*Sep 29 14:57:13.886: %SYS-6-CPURESFALLING: System is no longer seeing
global high cpu at interrupt level for the configured critical limit 70%,
current value 44%
============================================================================
===
============================================================================
===
PE-Router#sho proc cpu sort 5min
CPU utilization for five seconds: 1%/1%; one minute: 3%; five minutes: 6%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
436 1566429721118606284 140 0.00% 0.35% 0.35% 0 Port manager
per
7 205461504 12733776 16135 0.00% 0.33% 0.32% 0 Check heaps
433 12936 87590 147 0.15% 0.01% 0.18% 1 Virtual Exec
334 93262848 29028603 3212 0.00% 0.14% 0.14% 0 HIDDEN VLAN
Proc
200 40238324 301738465 133 0.00% 0.13% 0.13% 0 IP Input
202 2747572 19557286 140 0.00% 0.02% 0.04% 0 ADJ resolve
proc
55 31762204 999074 31791 0.00% 0.03% 0.03% 0 Per-minute
Jobs
289 25288024 27364507 924 0.00% 0.03% 0.03% 0 IPC LC
Message H
486 11390424 313585288 36 0.00% 0.03% 0.01% 0 TDP Hello
25 20877464 86502304 241 0.00% 0.03% 0.01% 0 IPC Seat
Manager
104 11541792 46704022 247 0.00% 0.01% 0.01% 0
esw_vlan_stat_pr
313 16750072 85168825 196 0.00% 0.02% 0.00% 0 CEF: IPv4
proces
============================================================================
===
I followed the troubleshooting high CPU Utilization Guide but still cannot
detect which kind of traffic is punted to CPU. Can anyone in the list show
me the way how to detect which packets sending to CPU on 7609 router?
Here are the troubleshooting guide that I have checked:
http://www.cisco.com/en/US/products/hw/routers/ps368/prod_tech_notes_list.ht
ml#anchor4
http://www.cisco.com/en/US/products/hw/switches/ps708/products_tech_note0918
6a00804916e0.shtml#utilities
Thanks,
Rin
More information about the cisco-nsp
mailing list