[c-nsp] Cisco 3750 high CPU utilization HL3U bkgrd]
Ioan Branet
ioan.branet at gmail.com
Wed Apr 1 06:29:14 EDT 2009
Hello All,
Have you encountered a similar situation?
Thank you,
On Tue, Mar 31, 2009 at 3:43 PM, Ioan Branet <ioan.branet at gmail.com> wrote:
> Hello,
>
> We have many Cisco 3750 switches in our network which have high CPU
> utilization.It seems that the process that cause this high load is:HL3U
> bkgrd process.
>
> The problem is solved after a reload but appears again after 3-4 months.
>
> We changed also the IOS but with no results.
>
> It seems that it is a bug but I am not very sure.
>
>
>
> sh processes cpu sorted | ex 0.00
> CPU utilization for five seconds: 99%/28%; one minute: 85%; five minutes:
> 81%
> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> 108 389775804 4389443 88799 57.57% 40.01% 39.39% 0 HL3U bkgrd
> proce
> 58 11854779 72185839 164 3.50% 2.77% 2.31% 0 HLFM address
> lea
> 292 689 192 3588 1.91% 0.33% 0.07% 1 Virtual
> Exec 47 12845296 2142151 5996 1.11% 1.00% 1.04% 0 FE
> free chunk 245 17376827 532655 32623 0.63% 0.51% 0.52% 0
> MFI LFD Stats Pr
> 107 5476276 58476944 93 0.63% 0.62% 0.58% 0 Hulc LED
> Process
> 74 768210 21312879 36 0.31% 0.09% 0.08% 0 hpm main
> process
> 135 6540410 20282165 322 0.15% 0.18% 0.22% 0 IP
> Input 143 3566619 27781902 128 0.15% 0.24% 0.20% 0
> Spanning Tree 45 1004640 128285520 7 0.15% 0.15% 0.13%
> 0 Fifo Error Detec
> 138 1152329 2735155 421 0.15% 0.13% 0.12% 0 PI MATM Aging
> Pr
>
>
> sh version
> Cisco IOS Software, C3750ME Software (C3750ME-I5-M), Version 12.2(37)SE1,
> RELEASE SOFTWARE (fc1)
> Copyright (c) 1986-2007 by Cisco Systems, Inc.
> Compiled Thu 05-Jul-07 20:06 by antonino
> Image text-base: 0x00003000, data-base: 0x0163F400
>
> ROM: Bootstrap program is C3750 boot loader
> BOOTLDR: C3750ME Boot Loader (C3750ME-HBOOT-M) Version 12.1(14r)AX, RELEASE
> SOFTWARE (fc1)
>
> vic102 uptime is 4 weeks, 4 days, 10 hours, 9 minutes
> System returned to ROM by power-on
> System restarted at 03:01:54 GMT Fri Feb 27 2009
> System image file is "flash:c3750me-i5-mz.122-37.SE1.bin"
>
> cisco ME-C3750-24TE (PowerPC405) processor (revision F0) with
> 118784K/12280K bytes of memory.
> Processor board ID CAT1043NM05
> Last reset from power-on
> 8 Virtual Ethernet interfaces
> 24 FastEthernet interfaces
> 4 Gigabit Ethernet interfaces
> The password-recovery mechanism is enabled.
>
> 1024K bytes of flash-simulated non-volatile configuration memory.
> Base ethernet MAC Address : 00:19:E8:87:23:00
> Motherboard assembly number : 73-9938-04
> Motherboard serial number : CAT104356B7
> Model revision number : F0
> Motherboard revision number : A0
> Model number : ME-C3750-24TE-M
> Daughterboard assembly number : 73-9939-02
> Daughterboard serial number : CAT104355CQ
> System serial number : CAT1043NM05
> Top Assembly Part Number : 800-25952-04
> Top Assembly Revision Number : C0
> Version ID : V05
> CLEI Code Number : COM1510ARA
> Daughterboard revision number : A0
> Hardware Board Revision Number : 0x09
>
>
> Switch Ports Model SW Version SW
> Image ------ ----- ----- ----------
> ---------- * 1 28 ME-C3750-24TE
> 12.2(37)SE1 C3750ME-I5-M
> Configuration register is 0xF
>
> #sh memory | i HL
> 030C8118 0000005000 030C804C 030C94CC 001 -------- -------- 005CF8E4 HLFM
> MAC
> 030C94CC 0000000808 030C8118 030C9820 001 -------- -------- 005CF93C HLFM
> IP
> 0320A434 0000000808 0320A008 0320A788 001 -------- -------- 00CB2F74
> HL3U_IPV4_TABLE_CHUNK
> 0320A788 0000020000 0320A434 0320F5D4 001 -------- -------- 00CB2F9C
> HL3U_FIB_TYPE_CHUNK
> 0320F5D4 0000032768 0320A788 03217600 001 -------- -------- 00CB2FC4
> HL3U_MPATH_ADJ_TYPE_CHUNK
> 03217600 0000000808 0320F5D4 03217954 001 -------- -------- 00CB2FEC
> HL3U_FIB_WITH_ADJ_OR_TCAM_FAIL_CHUNK
> 03217954 0000002000 03217600 03218150 001 -------- -------- 00CB3014
> HL3U_COVERING_FIB_CHUNK
> 03218150 0000000808 03217954 032184A4 001 -------- -------- 00CB303C
> HL3U_ARP_HRPC_THROTTLE_CHUNKS
> 032184A4 0000000432 03218150 03218680 001 -------- -------- 00CB3064
> HL3U_HSRP_RETRY_CHUNKS
> 03218680 0000000808 032184A4 032189D4 001 -------- -------- 00CB308C
> HL3U_PROXY_ARP_CHUNKS
> 032189D4 0000000432 03218680 03218BB0 001 -------- -------- 00CB30B4
> HL3U_QUERIER_INFO_CHUNKS
> 03218BB0 0000003620 032189D4 03219A00 001 -------- -------- 00CB30DC
> HL3U_ICMP_REDIRECT_Q_CHUNK
> 03219A00 0000000296 03218BB0 03219B54 001 -------- -------- 00CB3104
> HL3U_OUT_ACL_FULL_CHUNKS
> 032F2BCC 0000000960 032F252C 032F2FB8 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 0330BD38 0000000176 0330B698 0330BE14 001 -------- -------- 00B18A5C
> HL2MCM
> 0330C174 0000000160 0330C0C0 0330C240 001 -------- -------- 01622A8C
> HL2MCM
> 036F9C6C 0000000972 036F9A8C 036FA064 001 -------- -------- 00CB7108
> HL3U_FIB_WITH_
> 036FA064 0000000872 036F9C6C 036FA3F8 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 0390629C 0000000024 03906258 039062E0 001 -------- -------- 00B1AFAC
> HL2MCM
> 039906D8 0000001292 0399008C 03990C10 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03993CC4 0000000808 03993690 03994018 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 0399C258 0000001476 0399BC14 0399C848 001 -------- -------- 00CB7108
> HL3U_FIB_WITH_
> 0399C964 0000005000 0399C848 0399DD18 001 -------- -------- 005CB1E0 HLFM
> MAC
> 03A0A610 0000001096 03A0A3EC 03A0AA84 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03A2B8E4 0000000808 03A2B858 03A2BC38 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03A2BC38 0000000872 03A2B8E4 03A2BFCC 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03A2E864 0000007768 03A2E624 03A306E8 001 -------- -------- 005CB1E0 HLFM
> MAC
> 03A30D28 0000000808 03A30CBC 03A3107C 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03A92BD4 0000000808 03A92590 03A92F28 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03EE36A8 0000005000 03EE3634 03EE4A5C 001 -------- -------- 005CB1E0 HLFM
> MAC
> 03F24DCC 0000001008 03F24D54 03F251E8 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03F43C34 0000008200 03F43BE8 03F45C68 001 -------- -------- 005CB1E0 HLFM
> MAC
> 03F4C260 0000000808 03F4C1CC 03F4C5B4 001 -------- -------- 005CB5B8 HLFM
> IP
> 03F85410 0000000808 03F84DBC 03F85764 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03F85EF0 0000001008 03F85764 03F8630C 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03F872A8 0000001288 03F86F08 03F877DC 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03FEBD34 0000000808 03FEB5A8 03FEC088 001 -------- -------- 00CB7108
> HL3U_FIB_WITH_
> 03FEC088 0000001076 03FEBD34 03FEC4E8 001 -------- -------- 00CB7108
> HL3U_FIB_WITH_
> 03FEEF64 0000000808 03FEE7D8 03FEF2B8 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03FEF2B8 0000000832 03FEEF64 03FEF624 001 -------- -------- 00CBE090
> HL3U_FIB_WITH_
> 03FF05F0 0000005944 03FF04FC 03FF1D54 001 -------- -------- 005CB1E0 HLFM
> MAC
>
>
> #sh sdm prefer
> The current template is "desktop routing" template.
> The selected template optimizes the resources in
> the switch to support this level of features for
> 8 routed interfaces and 1024 VLANs.
>
> number of unicast mac addresses: 3K
> number of IPv4 IGMP groups + multicast routes: 1K
> number of IPv4 unicast routes: 11K
> number of directly-connected IPv4 hosts: 3K
> number of indirect IPv4 routes: 8K
> number of IPv4 policy based routing aces: 0.5K
> number of IPv4/MAC qos aces: 0.5K
> number of IPv4/MAC security aces: 1K
>
> Mar 31 12:48:11: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:48:17: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:48:28: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:48:46: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:49:00: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:49:47: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:50:39: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:51:02: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:51:34: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:51:55: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:53:11: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:53:25: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:53:37: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:54:36: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:56:44: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 12:57:48: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 13:02:22: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 13:02:48: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 13:03:56: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 13:07:31: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
> Mar 31 13:30:37: %OCE-3-MISSING_HANDLER_FOR_SW_OBJ_TYPE: Missing handler
> for 'non choice oce get next' function for type Midchain
>
>
> On other 3750 from our network:
>
> sh processes cpu sorted | ex 0.00
> CPU utilization for five seconds: 43%/2%; one minute: 58%; five minutes:
> 59%
> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> 108 2635783702 47442165 55558 30.55% 48.37% 49.28% 0 HL3U bkgrd
> proce
> 294 500 2692 185 3.19% 0.55% 0.13% 1 Virtual
> Exec 47 296813504 43714587 6789 1.43% 1.70% 1.74% 0 FE
> free chunk 107 58156203 265066803 219 0.63% 0.42% 0.37% 0
> Hulc LED Process
> 247 88726248 2800468 31682 0.47% 0.45% 0.47% 0 MFI LFD Stats
> Pr
> 117 3310849 6664259 496 0.15% 0.04% 0.03% 0 HRPC qos
> request
> 135 8065160 45776840 176 0.15% 0.07% 0.04% 0 IP
> Input 243 28572622 56335656 507 0.15% 0.17% 0.18% 0
> ISIS Upd
> #sh ver
> Cisco IOS Software, C3750ME Software (C3750ME-I5-M), Version 12.2(37)SE1,
> RELEASE SOFTWARE (fc1)
> Copyright (c) 1986-2007 by Cisco Systems, Inc.
> Compiled Thu 05-Jul-07 20:06 by antonino
> Image text-base: 0x00003000, data-base: 0x0163F400
>
> ROM: Bootstrap program is C3750 boot loader
> BOOTLDR: C3750ME Boot Loader (C3750ME-HBOOT-M) Version 12.1(14r)AX, RELEASE
> SOFTWARE (fc1)
>
> rom101 uptime is 27 weeks, 5 days, 18 hours, 13 minutes
> System returned to ROM by power-on
> System restarted at 20:03:10 CETDST Wed Sep 17 2008
> System image file is "flash:c3750me-i5-mz.122-37.SE1.bin"
>
> cisco ME-C3750-24TE (PowerPC405) processor (revision F0) with
> 118784K/12280K bytes of memory.
> Processor board ID CAT1111NLH3
> Last reset from power-on
> 3 Virtual Ethernet interfaces
> 24 FastEthernet interfaces
> 4 Gigabit Ethernet interfaces
> The password-recovery mechanism is enabled.
>
> 1024K bytes of flash-simulated non-volatile configuration memory.
> Base ethernet MAC Address : 00:1B:2B:E6:4B:00
> Motherboard assembly number : 73-9938-04
> Motherboard serial number : CAT11115HNX
> Model revision number : F0
> Motherboard revision number : B0
> Model number : ME-C3750-24TE-M
> Daughterboard assembly number : 73-9939-02
> Daughterboard serial number : CAT11115KVD
> System serial number : CAT1111NLH3
> Top Assembly Part Number : 800-25952-04
> Top Assembly Revision Number : C0
> Version ID : V05
> CLEI Code Number : COM1510ARA
> Daughterboard revision number : A0
> Hardware Board Revision Number : 0x09
>
>
> Switch Ports Model SW Version SW
> Image ------ ----- ----- ----------
> ---------- * 1 28 ME-C3750-24TE
> 12.2(37)SE1 C3750ME-I5-M
> Configuration register is 0xF
> Do you have any idea what is the root cause of this issue?
>
>
>
>
> Thnak you,
>
> --
> Ioan Branet
>
>
--
Ioan Branet
CCIE #23474 R&S
More information about the cisco-nsp
mailing list