I think Sup1 will show # of entries. Here is the output of show mls on a Sup1A:
1201_ADC1_L2> (enable) sho mls
Total packets switched = 2979494950
Total Active MLS entries = 665
IP Multilayer switching aging time = 256 seconds
IP Multilayer switching fast aging time = 0 seconds, packet threshold = 0
IP Current flow mask is Destination flow
Active IP MLS entries = 665
Netflow Data Export version: 7
Netflow Data Export disabled
Netflow Data Export port/host is not configured.
Total packets exported = 0
I am graphing CPU usage and I see a gradually increasing trend until after NBAR was removed.
Yes, CEF is enabled on the MSFC.
Anchi
-----Original Message-----
From: Jason LeBlanc [mailto:jml@packetpimp.org]
Sent: Thursday, May 23, 2002 1:36 PM
To: Zhang, Anchi; David Sinn
Cc: cisco-nsp@puck.nether.net
Subject: RE: [nsp] what uses all the cpu cycles
Should show # of entries in the table. This may or may not be the problem,
or only a partial contributor. If it is, analyzing your traffic patterns
and adjusting your aging can help A LOT. As an example, using MLS on a
device that handles 'internet' traffic (lots of dests) will pound one of
these things, getting the MLS table very full. Cisco recommends tuning such
that no more than 32k entries are in the table, more than this can lead to
hash collisions as new flows are added to the table, even though Cisco says
this, I've seen 170k entries when the table is supposedly limited to 128k
entries, take this with a grain of salt. =P If the packet can't use a
shortcut in MLS based on an entry in the table, it hits the MSFC CPU to be
routed. This is an issue with lots of dests and lots of pps, not so much so
in other situations. Since you're running IPX my experience is less useful
to you, but the architectural limits do still apply. Are you graphing MLS
and CPU? If not, do so and look for trends. If the trend is traffic
related you'll see it, and MLS or some other limitation with the type of
packet/config/traffic is likely at least contributing. CEF is enabled on
MSFC?
-----Original Message-----
From: Zhang, Anchi [mailto:AZhang@reliant.com]
Sent: Thursday, May 23, 2002 11:23 AM
To: Jason LeBlanc; David Sinn
Cc: cisco-nsp@puck.nether.net
Subject: RE: [nsp] what uses all the cpu cycles
REP_Dist1_L2> (enable) sho mls
Total packets switched = 248124103039
Total bytes switched = 179854738474984
Total routes = 5144
IP statistics flows aging time = 256 seconds
IP statistics flows fast aging time = 0 seconds, packet threshold = 0
IP Current flow mask is Destination flow
Netflow Data Export version: 8
Netflow Data Export disabled
Netflow Data Export port/host is not configured.
Total packets exported = 0
IPX statistics flows aging time = 256 seconds
IPX flow mask is Destination flow
IPX max hop is 255
-----Original Message-----
From: Jason LeBlanc [mailto:jml@packetpimp.org]
Sent: Thursday, May 23, 2002 12:20 PM
To: Zhang, Anchi; David Sinn
Cc: cisco-nsp@puck.nether.net
Subject: RE: [nsp] what uses all the cpu cycles
Running L3? What is your MLS aging?
-----Original Message-----
From: Zhang, Anchi [mailto:AZhang@reliant.com]
Sent: Thursday, May 23, 2002 9:16 AM
To: David Sinn
Cc: cisco-nsp@puck.nether.net
Subject: RE: [nsp] what uses all the cpu cycles
NBAR was removed last night and the CPU usage on both MSFCs has been much
lower. It has also been determined that IPX SAP traffic is responsible for
bursty high traffic and the corresponding high CPU. Here is one capture of
today's CPU usage at its highest:
REP29_Dist1_L3#sho proc cpu | inc CPU | 80 | 81 | 82
CPU utilization for five seconds: 51%/11%; one minute: 25%; five minutes:
23%
80 1080880002877932016 37 4.58% 0.01% 0.60% 0 IPX Input
81 71420700 372904668 191 2.03% 0.35% 0.33% 0 IPX RIP In
82 8411216162327729917 361 26.52% 5.35% 4.85% 0 IPX SAP In
Many thanks to all, especially David Sinn, for the great help.
Anchi
-----Original Message-----
From: Zhang, Anchi
Sent: Wednesday, May 22, 2002 5:03 PM
To: David Sinn
Cc: cisco-nsp@puck.nether.net
Subject: RE: [nsp] what uses all the cpu cycles
I will try to persuade the management to remove the NBAR at least as a
process of elimination.
What still bothers me is that the MSFC that serves as the HSRP standby also
shows high CPU at the same time even though its input rates on its
interfaces are much lower.
Anchi
-----Original Message-----
From: David Sinn [mailto:dsinn@microsoft.com]
Sent: Wednesday, May 22, 2002 4:27 PM
To: Zhang, Anchi
Subject: RE: [nsp] what uses all the cpu cycles
That is probably it then. It takes the MSFC to delve deep enough in to
the packet to make NBAR usable.
David
-----Original Message-----
From: Zhang, Anchi [mailto:AZhang@reliant.com]
Sent: Wednesday, May 22, 2002 2:19 PM
To: David Sinn
Subject: RE: [nsp] what uses all the cpu cycles
But the heavy traffic is most inbound on other interfaces with NBAR in:
Vlan120 is up, line protocol is up
5 minute input rate 59243000 bits/sec, 5215 packets/sec
5 minute output rate 45000 bits/sec, 27 packets/sec
Vlan142 is up, line protocol is up
5 minute input rate 1091000 bits/sec, 783 packets/sec
5 minute output rate 3000 bits/sec, 3 packets/sec
Vlan120 is up, line protocol is up
5 minute input rate 59243000 bits/sec, 5215 packets/sec
5 minute output rate 45000 bits/sec, 27 packets/sec
Vlan142 is up, line protocol is up
5 minute input rate 1091000 bits/sec, 783 packets/sec
5 minute output rate 3000 bits/sec, 3 packets/sec
Vlan143 is up, line protocol is up
5 minute input rate 24385000 bits/sec, 3474 packets/sec
5 minute output rate 93000 bits/sec, 124 packets/sec
Vlan182 is up, line protocol is up
5 minute input rate 4944000 bits/sec, 613 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
Vlan196 is up, line protocol is up
5 minute input rate 5841000 bits/sec, 927 packets/sec
5 minute output rate 4632000 bits/sec, 815 packets/sec
Vlan197 is up, line protocol is up
5 minute input rate 5376000 bits/sec, 945 packets/sec
5 minute output rate 5714000 bits/sec, 890 packets/sec
Vlan250 is up, line protocol is up
5 minute input rate 12208000 bits/sec, 1835 packets/sec
5 minute output rate 3000 bits/sec, 1 packets/sec
-----Original Message-----
From: David Sinn [mailto:dsinn@microsoft.com]
Sent: Wednesday, May 22, 2002 4:13 PM
To: Zhang, Anchi
Subject: RE: [nsp] what uses all the cpu cycles
NBAR is MSFC processed, but you have a inbound policy and on VLAN 10 you
are switching packets out. As such I wouldn't think that it is the
cause.
I would suggest contact Cisco to see if there is a bug. I've had one of
my systems do the same thing and they are still trying to track down
what the cause it.
David
-----Original Message-----
From: Zhang, Anchi [mailto:AZhang@reliant.com]
Sent: Wednesday, May 22, 2002 1:55 PM
To: David Sinn
Subject: RE: [nsp] what uses all the cpu cycles
They are sup2s:
REP_Dist1_L2> (enable) sho mod 1-2
Mod Slot Ports Module-Type Model Sub Status
--- ---- ----- ------------------------- ------------------- ---
--------
1 1 2 1000BaseX Supervisor WS-X6K-SUP2-2GE yes ok
2 2 2 1000BaseX Supervisor WS-X6K-SUP2-2GE yes standby
Mod Module-Name Serial-Num
--- ------------------- -----------
1 SAD05040AY0
2 SAD05040X3Z
Mod MAC-Address(es) Hw Fw Sw
--- -------------------------------------- ------ ----------
-----------------
1 00-01-c9-da-05-96 to 00-01-c9-da-05-97 2.2 6.1(3) 6.2(2)
00-01-c9-da-05-94 to 00-01-c9-da-05-95
00-d0-02-af-cc-00 to 00-d0-02-af-cf-ff
2 00-01-c9-da-04-92 to 00-01-c9-da-04-93 2.2 6.1(3) 6.2(2)
00-01-c9-da-04-90 to 00-01-c9-da-04-91
Mod Sub-Type Sub-Model Sub-Serial Sub-Hw
--- ----------------------- ------------------- ----------- ------
1 L3 Switching Engine II WS-F6K-PFC2 SAD05040YW4 1.3
2 L3 Switching Engine II WS-F6K-PFC2 SAD050310LB 1.3
I do have NBAR on each interface. Maybe that is it?
class-map match-any http-hacks
match protocol http url "*.ida*"
match protocol http url "*cmd.exe*"
match protocol http url "*root.exe*"
match protocol http url "*readme.eml*"
match protocol http url "*sample.eml"
match protocol http url "*desktop.eml"
match protocol http url "*readme.exe"
match protocol http url "*Readme.eml*"
match protocol http url "*Readme.exe*"
match protocol http url "*PUTA!!.SCR*"
match protocol http url "*PUTA!!.EML*"
match protocol http url "*PUTA!!.eml*"
match protocol http url "*PUTA!!.scr*"
match protocol http url "*puta!!.scr*"
match protocol http url "*puta!!.eml*"
!
policy-map mark-inbound-http-hacks
class http-hacks
set ip dscp 1
!
interface Vlan10
ip address 10.51.10.254 255.255.255.0
ip helper-address 10.51.100.100
ip helper-address 10.51.101.100
no ip redirects
ip pim sparse-mode
carrier-delay msec 0
ipx network 5600029
service-policy input mark-inbound-http-hacks
-----Original Message-----
From: David Sinn [mailto:dsinn@microsoft.com]
Sent: Wednesday, May 22, 2002 3:45 PM
To: Zhang, Anchi
Subject: RE: [nsp] what uses all the cpu cycles
Your .txt files definitely back up that your MSFC is switching a bunch
of packets, and more troublesome it is switching it in FAST/Cache'd
mode. This means that the PFC SHOULD have been able to do it.
Is this a Sup2, or Sup1A? Do you have ACL's on any of the interfaces,
and if so are they long (hundreds of lines)?
If you overflow the PFC's ACL space, the PFC will punt packets to the
MSFC for switching since it can't make the ACL decision.
If you are on a Spu2, I would suggest running a "sho ip cef sum" on the
MSFC and a "sho mls cef" on the Sup and verify that they both have the
same number of prefixes (or close if you have 'normal' routing changes
in your network.
If this is a Sup1A, then you may have too many flows, and you can "fix"
the problem by changing your flow cache ager to a lower value. It
doesn't actually prevent the problem, it just gives you more head room.
David
-----Original Message-----
From: Zhang, Anchi [mailto:AZhang@reliant.com]
Sent: Wednesday, May 22, 2002 1:32 PM
To: David Sinn
Subject: RE: [nsp] what uses all the cpu cycles
Many thanks. -anchi
-----Original Message-----
From: David Sinn [mailto:dsinn@microsoft.com]
Sent: Wednesday, May 22, 2002 3:30 PM
To: Zhang, Anchi; cisco-nsp@puck.nether.net
Subject: RE: [nsp] what uses all the cpu cycles
Your MSFC is switching a bunch of packets (64%), and then you have
somewhere around 30% doing process level things. The output of the
"show proc cpu" will tell you which set of processes are causing that
30% load (which sounds a little high).
I'd be happy to take a look at what you have captured so far if you
want.
David
-----Original Message-----
From: Zhang, Anchi [mailto:AZhang@reliant.com]
Sent: Wednesday, May 22, 2002 1:12 PM
To: cisco-nsp@puck.nether.net
Subject: [nsp] what uses all the cpu cycles
Greetings,
Several times a day two of my MSFC2s/6509s show high CPU usage:
CPU utilization for five seconds: 91%/64%; one minute: 89%; five
minutes: 86%
One MSFC2 serves mostly as an HSRP standby for the other.
I captured the output of the commands below during low CPU and during
high CPU but was unable to figure out what contributed to the high CPU.
show processes cpu
show interfaces
show interfaces switching
show interfaces stat
show align
show version
show log
If someone on the list is experienced in analyzing the CPU usage and is
willing to help, I can send the stats to him/her.
Anchi
This archive was generated by hypermail 2b29 : Sun Aug 04 2002 - 04:13:45 EDT