[c-nsp] Cisco 3750 High CPU

Chris Lane clane1875 at gmail.com
Wed Apr 7 17:08:06 EDT 2010


we do not have a spanning tree instance
sh spanning-tree detail

No spanning tree instance exists.



On Wed, Apr 7, 2010 at 3:37 PM, <cisco-nsp-request at puck.nether.net> wrote:

> Send cisco-nsp mailing list submissions to
>        cisco-nsp at puck.nether.net
>
> To subscribe or unsubscribe via the World Wide Web, visit
>        https://puck.nether.net/mailman/listinfo/cisco-nsp
> or, via email, send a message with subject or body 'help' to
>        cisco-nsp-request at puck.nether.net
>
> You can reach the person managing the list at
>        cisco-nsp-owner at puck.nether.net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of cisco-nsp digest..."
>
>
> Today's Topics:
>
>   1. Re: portchannel load balancing between L3 switch and router
>      (Arie Vayner (avayner))
>   2. Re: ISP Attack Discovery (Nick Voth)
>   3. Re: portchannel load balancing between L3 switch and router
>      (Pavel Dimow)
>   4. Cisco 3750 High CPU (Chris Lane)
>   5. Re: Cisco 3750 High CPU (Luan Nguyen)
>   6. Re: Cisco 3750 High CPU (Mack McBride)
>   7. Re: Cisco 3750 High CPU (David Prall)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 7 Apr 2010 18:07:11 +0200
> From: "Arie Vayner (avayner)" <avayner at cisco.com>
> To: "Pavel Dimow" <paveldimow at gmail.com>, <cisco-nsp at puck.nether.net>
> Subject: Re: [c-nsp] portchannel load balancing between L3 switch and
>        router
> Message-ID:
>        <FDD1CDB3FB499E4087CC7670BBCA22C601905616 at XMB-AMS-101.cisco.com>
> Content-Type: text/plain;       charset="iso-8859-1"
>
> Pavel,
>
> Is that the same SCE, or two separate SCEs?
> If it's the same one, then there is no problem. The SCE has a single global
> session state for all the ports.
>
> If these are separate SCEs, then you may want to use PBR (using source IP
> on one side and destination IP on the other side). This solution is called
> MGSCP (multi-gigabit service control point). In this case you would need to
> break the etherchannel and use L3 interfaces... I can provide more details
> if relevant.
>
> Arie
>
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net [mailto:
> cisco-nsp-bounces at puck.nether.net] On Behalf Of Pavel Dimow
> Sent: Wednesday, April 07, 2010 18:32
> To: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] portchannel load balancing between L3 switch and
> router
>
> The problem is that between ASR and 7600 is SCE, and as I don't have
> enough experience
> with it, the per flow load balancing can be a problem as I have found that
> SCE
> requires incoming and outgoing packets to use the same port, or I am
> missing something?
>
> On Tue, Apr 6, 2010 at 4:56 PM, Arie Vayner (avayner) <avayner at cisco.com>
> wrote:
> > Elmar,
> >
> > Please check this link:
> > http://www.cisco.com/en/US/docs/ios/ios_xe/lanswitch/configuration/guide
> > /lsw_cfg_flwload_xe_ps9587_TSD_Products_Configuration_Guide_Chapter.html
> >
> > You should be looking at the "Flow Based" option...
> >
> > Per the command reference
> > (http://www.cisco.com/en/US/docs/ios/interface/command/reference/ir_l1.h
> > tml#wp1022040) This is available since IOS XE 2.5
> >
> > Arie
> >
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces at puck.nether.net
> > [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Elmar K. Bins
> > Sent: Tuesday, April 06, 2010 17:13
> > To: Pavel Dimow
> > Cc: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] portchannel load balancing between L3 switch and
> > router
> >
> > Re Pavel,
> >
> > paveldimow at gmail.com (Pavel Dimow) wrote:
> >
> >> port-channel load-balancing vlan-manual
> >>
> >> and I understand that I can't have one vlan loadbalanced across two
> >> portchannel members but
> >> only the one at the time and second member is standby.
> >>
> >> Am I right or there is another solution?
> >
> > I have tried the same thing (trunk-channeling), and it just does not
> > work on ASRs (yet?).
> >
> > So yes, you are right, there seems to be no other solution right now.
> > Maybe someone from Cisco can tell us whether this issue will be
> > remedied.
> >
> > Yours,
> > ? ? ? ?Elmar.
> > _______________________________________________
> > cisco-nsp mailing list ?cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 07 Apr 2010 10:58:20 -0600
> From: Nick Voth <nvoth at estreet.com>
> To: <cisco-nsp at puck.nether.net>
> Subject: Re: [c-nsp] ISP Attack Discovery
> Message-ID: <C7E215CC.42CBC%nvoth at estreet.com<C7E215CC.42CBC%25nvoth at estreet.com>
> >
> Content-Type: text/plain;       charset="US-ASCII"
>
> One of the most useful tools we use all the time is a packet sniffer.
> Wireshark is great:
>
> http://www.wireshark.org/
>
> You would have to have a PC plugged in to a mirror port on your main switch
> in order to see all the packets.
>
> With the packet trace, you can get a good idea of who is sending the most
> traffic and where it's going. After you have that info, you have to have a
> firewall or switch/router ACL that can block the traffic. If it's a single
> source IP, (or a handful), that will work well. If the attack is
> distributed
> from hundreds or thousands of IP addresses, you really have to have a more
> intelligent device/firewall that can do session limiting, etc. That gets
> pretty complicated.
>
> Hope that helps.
>
> -Nick Voth
>
>
> > Message: 1
> > Date: Wed, 7 Apr 2010 15:49:21 +0000
> > From: sherif mostafa <sherifmka2004 at hotmail.com>
> > To: <cisco-nsp at puck.nether.net>
> > Subject: [c-nsp] ISP Attack Discovery
> > Message-ID: <BAY121-W2550986051311F7C40217DAD170 at phx.gbl>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> >
> >
> >
> > I'm working @ ISP and with our monitoring tools I sometimes find a large
> no.
> > of packet/secs which is most probably because of attack, scenario is that
> I've
> > large subnet for my ISP segmented into smaller subnets that are
> advertised to
> > three international providers, Question is:
> >
> >
> >
> >
> > How could I isolate the subnet which has the attack source IP ?
> > How could I know the source IP of the attacker directly ?
> > How to detect the attacker if the attack is from outside my ISP to an
> internal
> > IP ?
> > How could I investigate this issue ?
> >
> >
> >
> >
> >
> >
> > If anyone has experience in how to prevent or detect attacks and drop
> that
> > traffic please share knowledge with me..
> >
> >
> >
> > Thx.
>
>
>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 7 Apr 2010 19:25:21 +0200
> From: Pavel Dimow <paveldimow at gmail.com>
> To: "Arie Vayner (avayner)" <avayner at cisco.com>
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] portchannel load balancing between L3 switch and
>        router
> Message-ID:
>        <n2s6d2cb0d51004071025yf23019edl9b1e5057f5779c7a at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi Arie,
>
> those are 2 separate SCEs, and my first thought was to use
> etherchannel from the 7600 side I would
> use source mac lb and from the ASR to use dest mac lb (I have a couple
> of LNS's behind the 7600)
> Anyhow, because of ASR I can only use per vlan or per flow lb so MGSCP
> seams like the only candidate.
> If you can provide more details I would be very grateful.
>
> Thank you.
>
>
> On Wed, Apr 7, 2010 at 6:07 PM, Arie Vayner (avayner) <avayner at cisco.com>
> wrote:
> > Pavel,
> >
> > Is that the same SCE, or two separate SCEs?
> > If it's the same one, then there is no problem. The SCE has a single
> global session state for all the ports.
> >
> > If these are separate SCEs, then you may want to use PBR (using source IP
> on one side and destination IP on the other side). This solution is called
> MGSCP (multi-gigabit service control point). In this case you would need to
> break the etherchannel and use L3 interfaces... I can provide more details
> if relevant.
> >
> > Arie
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces at puck.nether.net [mailto:
> cisco-nsp-bounces at puck.nether.net] On Behalf Of Pavel Dimow
> > Sent: Wednesday, April 07, 2010 18:32
> > To: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] portchannel load balancing between L3 switch and
> router
> >
> > The problem is that between ASR and 7600 is SCE, and as I don't have
> > enough experience
> > with it, the per flow load balancing can be a problem as I have found
> that SCE
> > requires incoming and outgoing packets to use the same port, or I am
> > missing something?
> >
> > On Tue, Apr 6, 2010 at 4:56 PM, Arie Vayner (avayner) <avayner at cisco.com>
> wrote:
> >> Elmar,
> >>
> >> Please check this link:
> >>
> http://www.cisco.com/en/US/docs/ios/ios_xe/lanswitch/configuration/guide
> >> /lsw_cfg_flwload_xe_ps9587_TSD_Products_Configuration_Guide_Chapter.html
> >>
> >> You should be looking at the "Flow Based" option...
> >>
> >> Per the command reference
> >> (
> http://www.cisco.com/en/US/docs/ios/interface/command/reference/ir_l1.h
> >> tml#wp1022040) This is available since IOS XE 2.5
> >>
> >> Arie
> >>
> >>
> >> -----Original Message-----
> >> From: cisco-nsp-bounces at puck.nether.net
> >> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Elmar K. Bins
> >> Sent: Tuesday, April 06, 2010 17:13
> >> To: Pavel Dimow
> >> Cc: cisco-nsp at puck.nether.net
> >> Subject: Re: [c-nsp] portchannel load balancing between L3 switch and
> >> router
> >>
> >> Re Pavel,
> >>
> >> paveldimow at gmail.com (Pavel Dimow) wrote:
> >>
> >>> port-channel load-balancing vlan-manual
> >>>
> >>> and I understand that I can't have one vlan loadbalanced across two
> >>> portchannel members but
> >>> only the one at the time and second member is standby.
> >>>
> >>> Am I right or there is another solution?
> >>
> >> I have tried the same thing (trunk-channeling), and it just does not
> >> work on ASRs (yet?).
> >>
> >> So yes, you are right, there seems to be no other solution right now.
> >> Maybe someone from Cisco can tell us whether this issue will be
> >> remedied.
> >>
> >> Yours,
> >> ? ? ? ?Elmar.
> >> _______________________________________________
> >> cisco-nsp mailing list ?cisco-nsp at puck.nether.net
> >> https://puck.nether.net/mailman/listinfo/cisco-nsp
> >> archive at http://puck.nether.net/pipermail/cisco-nsp/
> >>
> >
> > _______________________________________________
> > cisco-nsp mailing list ?cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
>
>
>
> ------------------------------
>
> Message: 4
> Date: Wed, 7 Apr 2010 15:16:54 -0400
> From: Chris Lane <clane1875 at gmail.com>
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] Cisco 3750 High CPU
> Message-ID:
>        <p2x2e1cd851004071216p6967b6afi41af477d226a65b1 at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hello,
>
> I have all the sudden taken extremely high CPU:
> sh proc cpu sorted | e 0.0
> CPU utilization for five seconds: 99%/27%; one minute: 95%; five minutes:
> 92%
>  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
>  251     2985921     15274     195490 39.29% 11.05%  6.01%   0 hulc running
> con
>  171   630974187 249381380       2530 10.38%  9.27%  9.45%   0 Spanning
> Tree
>
>  117   133871232 301668428        443  4.63%  8.34%  9.47%   0 Hulc LED
> Process
>  68     3766455 374577924         10  4.15%  3.66%  3.20%   0 HLFM address
> lea
>  137   221859624  12599002      17609  2.39%  2.02%  2.05%   0 PI MATM
> Aging
> Pr
>  168   175580828 496683600        353  1.91%  3.89%  2.90%   0 IP Input
>
>  52     8324282 665636083         12  0.79%  0.43%  0.35%   0 Fifo Error
> Detec
>
> I know this isn't much but could anyone offer assistance?
>
> Thanks
> Chris
>
>
>
> --
> //CL
>
>
> ------------------------------
>
> Message: 5
> Date: Wed, 7 Apr 2010 15:26:07 -0400
> From: "Luan Nguyen" <luan at netcraftsmen.net>
> To: "'Chris Lane'" <clane1875 at gmail.com>, <cisco-nsp at puck.nether.net>
> Subject: Re: [c-nsp] Cisco 3750 High CPU
> Message-ID: <01b501cad688$2bdf47f0$839dd7d0$@net>
> Content-Type: text/plain;       charset="us-ascii"
>
> This link should provide some guidance regarding HULC running process.
>
> http://www.cisco.com/en/US/products/hw/switches/ps5023/products_tech_note091
> 86a00807213f5.shtml
>
>
> -Luan
>
>
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Chris Lane
> Sent: Wednesday, April 07, 2010 3:17 PM
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] Cisco 3750 High CPU
>
> Hello,
>
> I have all the sudden taken extremely high CPU:
> sh proc cpu sorted | e 0.0
> CPU utilization for five seconds: 99%/27%; one minute: 95%; five minutes:
> 92%
>  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
>  251     2985921     15274     195490 39.29% 11.05%  6.01%   0 hulc running
> con
>  171   630974187 249381380       2530 10.38%  9.27%  9.45%   0 Spanning
> Tree
>
>  117   133871232 301668428        443  4.63%  8.34%  9.47%   0 Hulc LED
> Process
>  68     3766455 374577924         10  4.15%  3.66%  3.20%   0 HLFM address
> lea
>  137   221859624  12599002      17609  2.39%  2.02%  2.05%   0 PI MATM
> Aging
> Pr
>  168   175580828 496683600        353  1.91%  3.89%  2.90%   0 IP Input
>
>  52     8324282 665636083         12  0.79%  0.43%  0.35%   0 Fifo Error
> Detec
>
> I know this isn't much but could anyone offer assistance?
>
> Thanks
> Chris
>
>
>
> --
> //CL
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
>
>
> ------------------------------
>
> Message: 6
> Date: Wed, 7 Apr 2010 12:32:46 -0700
> From: Mack McBride <mack.mcbride at viawest.com>
> To: Chris Lane <clane1875 at gmail.com>, "cisco-nsp at puck.nether.net"
>        <cisco-nsp at puck.nether.net>
> Subject: Re: [c-nsp] Cisco 3750 High CPU
> Message-ID:
>        <
> CCD664821F7A144CBE712A1DD4E6D94726D72B7AF9 at EXVMBX017-1.exch017.msoutlookonline.net
> >
>
> Content-Type: text/plain; charset="us-ascii"
>
> Based on the consistently high spanning-tree usage I would suspect a
> spanning tree issue.
> I suggest investigating the spanning-tree as well as looking at the logs.
>
> LR Mack McBride
> Network Architect
> Viawest, Inc.
>
>
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net [mailto:
> cisco-nsp-bounces at puck.nether.net] On Behalf Of Chris Lane
> Sent: Wednesday, April 07, 2010 1:17 PM
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] Cisco 3750 High CPU
>
> Hello,
>
> I have all the sudden taken extremely high CPU:
> sh proc cpu sorted | e 0.0
> CPU utilization for five seconds: 99%/27%; one minute: 95%; five minutes:
> 92%
>  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
>  251     2985921     15274     195490 39.29% 11.05%  6.01%   0 hulc running
> con
>  171   630974187 249381380       2530 10.38%  9.27%  9.45%   0 Spanning
> Tree
>
>  117   133871232 301668428        443  4.63%  8.34%  9.47%   0 Hulc LED
> Process
>  68     3766455 374577924         10  4.15%  3.66%  3.20%   0 HLFM address
> lea
>  137   221859624  12599002      17609  2.39%  2.02%  2.05%   0 PI MATM
> Aging
> Pr
>  168   175580828 496683600        353  1.91%  3.89%  2.90%   0 IP Input
>
>  52     8324282 665636083         12  0.79%  0.43%  0.35%   0 Fifo Error
> Detec
>
> I know this isn't much but could anyone offer assistance?
>
> Thanks
> Chris
>
>
>
> --
> //CL
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
>
> ------------------------------
>
> Message: 7
> Date: Wed, 7 Apr 2010 15:37:29 -0400
> From: "David Prall" <dcp at dcptech.com>
> To: "'Chris Lane'" <clane1875 at gmail.com>, <cisco-nsp at puck.nether.net>
> Subject: Re: [c-nsp] Cisco 3750 High CPU
> Message-ID: <008501cad689$c4dbc040$4e9340c0$@com>
> Content-Type: text/plain;       charset="us-ascii"
>
> I'd guess a spanning tree loop. The HULC process is what updates the pretty
> lights on the switch. So much is happening that it is having to change all
> the colors constantly.
>
> What other messages are you seeing.
>
> --
> http://dcp.dcptech.com
>
>
> > -----Original Message-----
> > From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-
> > bounces at puck.nether.net] On Behalf Of Chris Lane
> > Sent: Wednesday, April 07, 2010 3:17 PM
> > To: cisco-nsp at puck.nether.net
> > Subject: [c-nsp] Cisco 3750 High CPU
> >
> > Hello,
> >
> > I have all the sudden taken extremely high CPU:
> > sh proc cpu sorted | e 0.0
> > CPU utilization for five seconds: 99%/27%; one minute: 95%; five
> > minutes:
> > 92%
> >  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
> >  251     2985921     15274     195490 39.29% 11.05%  6.01%   0 hulc
> > running
> > con
> >  171   630974187 249381380       2530 10.38%  9.27%  9.45%   0 Spanning
> > Tree
> >
> >  117   133871232 301668428        443  4.63%  8.34%  9.47%   0 Hulc LED
> > Process
> >   68     3766455 374577924         10  4.15%  3.66%  3.20%   0 HLFM
> > address
> > lea
> >  137   221859624  12599002      17609  2.39%  2.02%  2.05%   0 PI MATM
> > Aging
> > Pr
> >  168   175580828 496683600        353  1.91%  3.89%  2.90%   0 IP Input
> >
> >   52     8324282 665636083         12  0.79%  0.43%  0.35%   0 Fifo
> > Error
> > Detec
> >
> > I know this isn't much but could anyone offer assistance?
> >
> > Thanks
> > Chris
> >
> >
> >
> > --
> > //CL
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
>
> ------------------------------
>
> _______________________________________________
> cisco-nsp mailing list
> cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
>
> End of cisco-nsp Digest, Vol 89, Issue 20
> *****************************************
>



-- 
//CL


More information about the cisco-nsp mailing list