[c-nsp] Cisco 3750 high CPU utilization HL3U bkgrd]
Ioan Branet
ioan.branet at gmail.com
Tue Mar 31 08:43:04 EDT 2009
Hello,
We have many Cisco 3750 switches in our network which have high CPU
utilization.It seems that the process that cause this high load is:HL3U
bkgrd process.
The problem is solved after a reload but appears again after 3-4 months.
We changed also the IOS but with no results.
It seems that it is a bug but I am not very sure.
sh processes cpu sorted | ex 0.00
CPU utilization for five seconds: 99%/28%; one minute: 85%; five minutes:
81%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
108 389775804 4389443 88799 57.57% 40.01% 39.39% 0 HL3U bkgrd
proce
58 11854779 72185839 164 3.50% 2.77% 2.31% 0 HLFM address
lea
292 689 192 3588 1.91% 0.33% 0.07% 1 Virtual
Exec 47 12845296 2142151 5996 1.11% 1.00% 1.04% 0 FE
free chunk 245 17376827 532655 32623 0.63% 0.51% 0.52% 0
MFI LFD Stats Pr
107 5476276 58476944 93 0.63% 0.62% 0.58% 0 Hulc LED
Process
74 768210 21312879 36 0.31% 0.09% 0.08% 0 hpm main
process
135 6540410 20282165 322 0.15% 0.18% 0.22% 0 IP
Input 143 3566619 27781902 128 0.15% 0.24% 0.20% 0
Spanning Tree 45 1004640 128285520 7 0.15% 0.15% 0.13%
0 Fifo Error Detec
138 1152329 2735155 421 0.15% 0.13% 0.12% 0 PI MATM Aging
Pr
sh version
Cisco IOS Software, C3750ME Software (C3750ME-I5-M), Version 12.2(37)SE1,
RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 05-Jul-07 20:06 by antonino
Image text-base: 0x00003000, data-base: 0x0163F400
ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750ME Boot Loader (C3750ME-HBOOT-M) Version 12.1(14r)AX, RELEASE
SOFTWARE (fc1)
vic102 uptime is 4 weeks, 4 days, 10 hours, 9 minutes
System returned to ROM by power-on
System restarted at 03:01:54 GMT Fri Feb 27 2009
System image file is "flash:c3750me-i5-mz.122-37.SE1.bin"
cisco ME-C3750-24TE (PowerPC405) processor (revision F0) with 118784K/12280K
bytes of memory.
Processor board ID CAT1043NM05
Last reset from power-on
8 Virtual Ethernet interfaces
24 FastEthernet interfaces
4 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
1024K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:19:E8:87:23:00
Motherboard assembly number : 73-9938-04
Motherboard serial number : CAT104356B7
Model revision number : F0
Motherboard revision number : A0
Model number : ME-C3750-24TE-M
Daughterboard assembly number : 73-9939-02
Daughterboard serial number : CAT104355CQ
System serial number : CAT1043NM05
Top Assembly Part Number : 800-25952-04
Top Assembly Revision Number : C0
Version ID : V05
CLEI Code Number : COM1510ARA
Daughterboard revision number : A0
Hardware Board Revision Number : 0x09
Switch Ports Model SW Version SW
Image ------ ----- ----- ----------
---------- * 1 28 ME-C3750-24TE
12.2(37)SE1 C3750ME-I5-M
Configuration register is 0xF
#sh memory | i HL
030C8118 0000005000 030C804C 030C94CC 001 -------- -------- 005CF8E4 HLFM
MAC
030C94CC 0000000808 030C8118 030C9820 001 -------- -------- 005CF93C HLFM
IP
0320A434 0000000808 0320A008 0320A788 001 -------- -------- 00CB2F74
HL3U_IPV4_TABLE_CHUNK
0320A788 0000020000 0320A434 0320F5D4 001 -------- -------- 00CB2F9C
HL3U_FIB_TYPE_CHUNK
0320F5D4 0000032768 0320A788 03217600 001 -------- -------- 00CB2FC4
HL3U_MPATH_ADJ_TYPE_CHUNK
03217600 0000000808 0320F5D4 03217954 001 -------- -------- 00CB2FEC
HL3U_FIB_WITH_ADJ_OR_TCAM_FAIL_CHUNK
03217954 0000002000 03217600 03218150 001 -------- -------- 00CB3014
HL3U_COVERING_FIB_CHUNK
03218150 0000000808 03217954 032184A4 001 -------- -------- 00CB303C
HL3U_ARP_HRPC_THROTTLE_CHUNKS
032184A4 0000000432 03218150 03218680 001 -------- -------- 00CB3064
HL3U_HSRP_RETRY_CHUNKS
03218680 0000000808 032184A4 032189D4 001 -------- -------- 00CB308C
HL3U_PROXY_ARP_CHUNKS
032189D4 0000000432 03218680 03218BB0 001 -------- -------- 00CB30B4
HL3U_QUERIER_INFO_CHUNKS
03218BB0 0000003620 032189D4 03219A00 001 -------- -------- 00CB30DC
HL3U_ICMP_REDIRECT_Q_CHUNK
03219A00 0000000296 03218BB0 03219B54 001 -------- -------- 00CB3104
HL3U_OUT_ACL_FULL_CHUNKS
032F2BCC 0000000960 032F252C 032F2FB8 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
0330BD38 0000000176 0330B698 0330BE14 001 -------- -------- 00B18A5C
HL2MCM
0330C174 0000000160 0330C0C0 0330C240 001 -------- -------- 01622A8C
HL2MCM
036F9C6C 0000000972 036F9A8C 036FA064 001 -------- -------- 00CB7108
HL3U_FIB_WITH_
036FA064 0000000872 036F9C6C 036FA3F8 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
0390629C 0000000024 03906258 039062E0 001 -------- -------- 00B1AFAC
HL2MCM
039906D8 0000001292 0399008C 03990C10 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03993CC4 0000000808 03993690 03994018 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
0399C258 0000001476 0399BC14 0399C848 001 -------- -------- 00CB7108
HL3U_FIB_WITH_
0399C964 0000005000 0399C848 0399DD18 001 -------- -------- 005CB1E0 HLFM
MAC
03A0A610 0000001096 03A0A3EC 03A0AA84 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03A2B8E4 0000000808 03A2B858 03A2BC38 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03A2BC38 0000000872 03A2B8E4 03A2BFCC 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03A2E864 0000007768 03A2E624 03A306E8 001 -------- -------- 005CB1E0 HLFM
MAC
03A30D28 0000000808 03A30CBC 03A3107C 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03A92BD4 0000000808 03A92590 03A92F28 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03EE36A8 0000005000 03EE3634 03EE4A5C 001 -------- -------- 005CB1E0 HLFM
MAC
03F24DCC 0000001008 03F24D54 03F251E8 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03F43C34 0000008200 03F43BE8 03F45C68 001 -------- -------- 005CB1E0 HLFM
MAC
03F4C260 0000000808 03F4C1CC 03F4C5B4 001 -------- -------- 005CB5B8 HLFM
IP
03F85410 0000000808 03F84DBC 03F85764 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03F85EF0 0000001008 03F85764 03F8630C 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03F872A8 0000001288 03F86F08 03F877DC 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03FEBD34 0000000808 03FEB5A8 03FEC088 001 -------- -------- 00CB7108
HL3U_FIB_WITH_
03FEC088 0000001076 03FEBD34 03FEC4E8 001 -------- -------- 00CB7108
HL3U_FIB_WITH_
03FEEF64 0000000808 03FEE7D8 03FEF2B8 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03FEF2B8 0000000832 03FEEF64 03FEF624 001 -------- -------- 00CBE090
HL3U_FIB_WITH_
03FF05F0 0000005944 03FF04FC 03FF1D54 001 -------- -------- 005CB1E0 HLFM
MAC
#sh sdm prefer
The current template is "desktop routing" template.
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.
number of unicast mac addresses: 3K
number of IPv4 IGMP groups + multicast routes: 1K
number of IPv4 unicast routes: 11K
number of directly-connected IPv4 hosts: 3K
number of indirect IPv4 routes: 8K
number of IPv4 policy based routing aces: 0.5K
number of IPv4/MAC qos aces: 0.5K
number of IPv4/MAC security aces: 1K
Mar 31 12:48:11: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:48:17: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:48:28: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:48:46: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:49:00: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:49:47: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:50:39: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:51:02: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:51:34: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:51:55: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:53:11: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:53:25: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:53:37: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:54:36: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:56:44: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 12:57:48: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 13:02:22: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 13:02:48: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 13:03:56: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 13:07:31: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
Mar 31 13:30:37: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler for
'non choice oce get next' function for type Midchain
On other 3750 from our network:
sh processes cpu sorted | ex 0.00
CPU utilization for five seconds: 43%/2%; one minute: 58%; five minutes: 59%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
108 2635783702 47442165 55558 30.55% 48.37% 49.28% 0 HL3U bkgrd
proce
294 500 2692 185 3.19% 0.55% 0.13% 1 Virtual
Exec 47 296813504 43714587 6789 1.43% 1.70% 1.74% 0 FE
free chunk 107 58156203 265066803 219 0.63% 0.42% 0.37% 0
Hulc LED Process
247 88726248 2800468 31682 0.47% 0.45% 0.47% 0 MFI LFD Stats
Pr
117 3310849 6664259 496 0.15% 0.04% 0.03% 0 HRPC qos
request
135 8065160 45776840 176 0.15% 0.07% 0.04% 0 IP
Input 243 28572622 56335656 507 0.15% 0.17% 0.18% 0
ISIS Upd
#sh ver
Cisco IOS Software, C3750ME Software (C3750ME-I5-M), Version 12.2(37)SE1,
RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 05-Jul-07 20:06 by antonino
Image text-base: 0x00003000, data-base: 0x0163F400
ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750ME Boot Loader (C3750ME-HBOOT-M) Version 12.1(14r)AX, RELEASE
SOFTWARE (fc1)
rom101 uptime is 27 weeks, 5 days, 18 hours, 13 minutes
System returned to ROM by power-on
System restarted at 20:03:10 CETDST Wed Sep 17 2008
System image file is "flash:c3750me-i5-mz.122-37.SE1.bin"
cisco ME-C3750-24TE (PowerPC405) processor (revision F0) with 118784K/12280K
bytes of memory.
Processor board ID CAT1111NLH3
Last reset from power-on
3 Virtual Ethernet interfaces
24 FastEthernet interfaces
4 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
1024K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:1B:2B:E6:4B:00
Motherboard assembly number : 73-9938-04
Motherboard serial number : CAT11115HNX
Model revision number : F0
Motherboard revision number : B0
Model number : ME-C3750-24TE-M
Daughterboard assembly number : 73-9939-02
Daughterboard serial number : CAT11115KVD
System serial number : CAT1111NLH3
Top Assembly Part Number : 800-25952-04
Top Assembly Revision Number : C0
Version ID : V05
CLEI Code Number : COM1510ARA
Daughterboard revision number : A0
Hardware Board Revision Number : 0x09
Switch Ports Model SW Version SW
Image ------ ----- ----- ----------
---------- * 1 28 ME-C3750-24TE
12.2(37)SE1 C3750ME-I5-M
Configuration register is 0xF
Do you have any idea what is the root cause of this issue?
Thnak you,
--
Ioan Branet
More information about the cisco-nsp
mailing list