[c-nsp] high cpu usage due to interrupts
Rodney Dunn
rodunn at cisco.com
Tue Feb 13 10:22:41 EST 2007
On Tue, Feb 13, 2007 at 04:07:49PM +0100, ulisses wrote:
> Hi Rodney
>
> On 2007-02-13 08:52:27, Rodney Dunn wrote:
> > It's an RSP1 and the latest today is an RSP16.
>
> heh :-)
>
> didn't take that into account:
>
> End-of-Life Announcement: September 24, 1999
>
> but is the load of 60000 pps a high load? for this hardware?
Could be. I haven't worked on one in 9 years. :)
Also depends on the features you have.
>
> > What you need to look at is the pps being switched through
> > the box as well as the features you have enabled.
>
> This is Cisco 7505, it seems that the backplane can handle 1.067 Gbps,
> and 1.1 Mpps, I suposse that also depends on the RSP...
> The information I found didn't mention to which RSP this information
> refers to.
:) That's the most important factor over dCEF.
>
> When not loaded, it is ~31000 packets/sec in 5 min average [2],
31kpps. That's probably normal CPU on that RP.
>
> If I scale this by 2 it becomes around 60000 pps, not a lot isn't it?
For that CPU it probably is.
>
> it seems there is a lot of margin...
>
> > The CPU level there is probably normal for that processor.
> >
> > If your VIPs have enough memory you could 'ip cef distributed'.
>
> 'ip cef distributed' is not an option completed by pressing the
> key '?' (only accounting option is shown, anyway i have entered it
> and the output of the "show ip interfaces | include switch" is
> the same output and the ratio of high cpu interrupts has not changed.
That code with that RP may not support it that far back.
>
> What I find extrange is that all percentage of the cpu usage goes
> to interrupts:
That's good. That tells you it's doing all the work switching packets
and not process work.
>
> Addionally (I haven't mentioned previously) is that when multicast
> comes in (around 10-15 Mbps) the cpu burns at 100%. But first
> I wanted to make clear that there is no problem with unicast traffic.
Oh yeah...that would burn it like a champ. Especially when you don't
have mdfs (multicast distributed fastswitching) capability.
And make sure even at the RP level you have "ip mroute-cache" enabled.
It used to not be on by default.
I think you need a newer RSP and VIPs that have the memory to do
dCEF/MDFS if you want to run those packet rates through a 75xx.
Rodney
>
> Thanks so much for your comments
>
> regards
>
> Ulisses
>
> [2]
>
> # show interfaces | include packets/sec
> 5 minute input rate 62946000 bits/sec, 8537 packets/sec
> 5 minute output rate 27570000 bits/sec, 7036 packets/sec
> 5 minute input rate 3234000 bits/sec, 1041 packets/sec
> 5 minute output rate 10653000 bits/sec, 1287 packets/sec
> 5 minute input rate 23616000 bits/sec, 5363 packets/sec
> 5 minute output rate 37962000 bits/sec, 5599 packets/sec
> 5 minute input rate 414000 bits/sec, 82 packets/sec
> 5 minute output rate 329000 bits/sec, 76 packets/sec
> 5 minute input rate 594000 bits/sec, 612 packets/sec
> 5 minute output rate 14166000 bits/sec, 1571 packets/sec
> 5 minute input rate 0 bits/sec, 0 packets/sec
> 5 minute output rate 0 bits/sec, 0 packets/sec
> 5 minute input rate 0 bits/sec, 0 packets/sec
> 5 minute output rate 324000 bits/sec, 76 packets/sec
>
> >
> >
> > Rodney
> >
> > On Tue, Feb 13, 2007 at 02:41:51PM +0100, ulisses wrote:
> > >
> > > Hi all
> > >
> > > I have followed the guidelines
> > >
> > > http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml
> > > http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml#high_cpu
> > >
> > > trying to find why I have a high cpu usage due to interrupts
> > > but I could not solve it, next I show diagnostics when only routing
> > > unicast traffic.
> > >
> > > The basic configuration of the system (Cisco 7500) is:
> > >
> > > IOS (tm) RSP Software (RSP-JSV-M), Version 12.1(14), RELEASE SOFTWARE (fc1)
> > > cisco RSP1 (R4600) processor with 65536K/2072K bytes of memory.
> > > R4600 CPU at 100Mhz, Implementation 32, Rev 2.0
> > > 2 AIP controllers (2 ATM).
> > > 2 VIP2 controllers (3 FastEthernet).
> > > 3 FastEthernet/IEEE 802.3 interface(s)
> > > 2 ATM network interface(s)
> > >
> > > This is the load of the router:
> > >
> > > EB-LasPalmas1#show interfaces | include is up|load
> > > ATM0/0 is up, line protocol is up
> > > reliability 255/255, txload 55/255, rxload 152/255
> > > ATM0/0.1 is up, line protocol is up
> > > reliability 255/255, txload 55/255, rxload 152/255
> > > FastEthernet1/0/0 is up, line protocol is up
> > > reliability 255/255, txload 82/255, rxload 16/255
> > > FastEthernet2/0/0 is up, line protocol is up
> > > reliability 255/255, txload 119/255, rxload 68/255
> > > FastEthernet3/0/0 is up, line protocol is up
> > > reliability 255/255, txload 1/255, rxload 1/255
> > > FastEthernet3/1/0 is up, line protocol is up
> > > reliability 255/255, txload 35/255, rxload 1/255
> > >
> > > This is the load near the peak load of the router. I also haven't
> > > seen any throttle in the interfaces.
> > >
> > > I think it is not too much, isn't it?
> > >
> > > I have enabled all possible swithing options possible, see [1] at the end
> > >
> > > I also have verified that there are no cache misses in the switching path
> > > and traffic uses the supossed switched path with show interfaces stats.
> > >
> > > The cpu load is 100% (47 of 48%) caused by interrupts:
> > >
> > > EB-LasPalmas1#show process cpu | include CPU|PID|Input
> > > CPU utilization for five seconds: 48%/47%; one minute: 45%; five minutes: 44%
> > > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> > > 13 176 1809 97 0.00% 0.00% 0.00% 0 ARP Input
> > > 40 7036 105293 66 0.00% 0.00% 0.00% 0 ATM OAM Input
> > > 44 6572080 1952147 3366 0.08% 0.09% 0.11% 0 IP Input
> > > 50 0 2 0 0.00% 0.00% 0.00% 0 SSCOP Input
> > > 56 0 2 0 0.00% 0.00% 0.00% 0 ILMI Input
> > > 64 0 2 0 0.00% 0.00% 0.00% 0 ATMSIG Input
> > > 69 0 1 0 0.00% 0.00% 0.00% 0 Probe Input
> > > 70 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
> > > 78 0 2 0 0.00% 0.00% 0.00% 0 SPX Input
> > > 83 964 926 1041 0.00% 0.00% 0.00% 0 Tag Input
> > > 87 0 1 0 0.00% 0.00% 0.00% 0 Net Input
> > > 100 7536 39990 188 0.00% 0.00% 0.00% 0 CLNS Input
> > > 102 28504 109557 260 0.00% 0.02% 0.00% 0 IGMP Input
> > >
> > >
> > > Following the guideline on high cpu usage cause interrupts I found that:
> > >
> > > Possible Causes of High CPU Utilization Due to Interrupts
> > > Inappropriate Switching Path ---------- Not likely IMHO
> > > CPU Performing Alignment Corrections -- No (verified)
> > > Router Overloaded with Traffic -------- No (verified)
> > > Software Bug -------------------------- I don't know (no access to Cisco TACS)
> > > Voice Ports Configured on the Router -- No
> > > Active Asynchronous Transfer Mode (ATM) Interfaces on the Router
> > > --------------------------------------- No (only the required ones)
> > > Many Parallel Express Forwarding (PXF) Punts to the RP
> > > -------------------------------------- No
> > >
> > > so I don't know what to check next... Any comment or suggestion?
> > >
> > > Thanks so much in advance
> > >
> > > Ulisses
> > >
> > >
> > > [1]
> > >
> > > EB-LasPalmas1#show ip interface | include is up|switch
> > > ATM0/0 is up, line protocol is up
> > > ATM0/0.1 is up, line protocol is up
> > > IP fast switching is enabled
> > > IP fast switching on the same interface is disabled
> > > IP Flow switching is enabled
> > > IP CEF switching is enabled
> > > IP Flow switching turbo vector
> > > IP Flow CEF switching turbo vector
> > > IP multicast fast switching is enabled
> > > IP multicast distributed fast switching is disabled
> > > FastEthernet1/0/0 is up, line protocol is up
> > > IP fast switching is enabled
> > > IP fast switching on the same interface is disabled
> > > IP Flow switching is enabled
> > > IP CEF switching is enabled
> > > IP Distributed switching is enabled
> > > IP Flow switching turbo vector
> > > IP Flow CEF switching turbo vector
> > > IP multicast fast switching is enabled
> > > IP multicast distributed fast switching is enabled
> > > FastEthernet2/0/0 is up, line protocol is up
> > > IP fast switching is enabled
> > > IP fast switching on the same interface is disabled
> > > IP Flow switching is enabled
> > > IP CEF switching is enabled
> > > IP Distributed switching is enabled
> > > IP Flow switching turbo vector
> > > IP Flow CEF switching turbo vector
> > > IP multicast fast switching is enabled
> > > IP multicast distributed fast switching is enabled
> > > FastEthernet3/0/0 is up, line protocol is up
> > > IP fast switching is enabled
> > > IP fast switching on the same interface is disabled
> > > IP Flow switching is enabled
> > > IP CEF switching is enabled
> > > IP Distributed switching is enabled
> > > IP Flow switching turbo vector
> > > IP Flow CEF switching turbo vector
> > > IP multicast fast switching is enabled
> > > IP multicast distributed fast switching is enabled
> > > FastEthernet3/1/0 is up, line protocol is up
> > > IP fast switching is enabled
> > > IP fast switching on the same interface is disabled
> > > IP Flow switching is enabled
> > > IP CEF switching is enabled
> > > IP Distributed switching is enabled
> > > IP Flow switching turbo vector
> > > IP Flow CEF switching turbo vector
> > > IP multicast fast switching is enabled
> > > IP multicast distributed fast switching is enabled
> > >
> > > --
> > > _______________________________ __ ___________________________________
> > > /_/
> > > Network Operation Center __ __ Email: noc at rediris.es
> > > RedIRIS/Red.es /_/ RedIRIS /_/
> > > Edificio Bronce __ Tel: + 34 912127625
> > > Pza. Manuel Gómez Moreno, s/n /_/ Urgencias: + 34 629148201
> > > E-28020 Madrid Fax: + 34 915568864
> > >
> > > _________________ Spanish Academic & Research Network _______________________
> > > _______________________________________________
> > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
>
> --
> _______________________________ __ ___________________________________
> /_/
> Network Operation Center __ __ Email: noc at rediris.es
> RedIRIS/Red.es /_/ RedIRIS /_/
> Edificio Bronce __ Tel: + 34 912127625
> Pza. Manuel Gómez Moreno, s/n /_/ Urgencias: + 34 629148201
> E-28020 Madrid Fax: + 34 915568864
>
> _________________ Spanish Academic & Research Network _______________________
More information about the cisco-nsp
mailing list