[c-nsp] high cpu usage due to interrupts

ulisses ulisses.alonso at rediris.es
Tue Feb 13 10:07:49 EST 2007


Hi Rodney

On 2007-02-13 08:52:27, Rodney Dunn wrote:
> It's an RSP1 and the latest today is an RSP16.

heh :-)

didn't take that into account:

End-of-Life Announcement: September 24, 1999

but is the load of 60000 pps a high load? for this hardware?

> What you need to look at is the pps being switched through
> the box as well as the features you have enabled.

This is Cisco 7505, it seems that the backplane can handle 1.067 Gbps,
and 1.1 Mpps, I suposse that also depends on the RSP...
The information I found didn't mention to which RSP this information 
refers to.

When not loaded, it is ~31000 packets/sec in 5 min average [2],

If I scale this by 2 it becomes around 60000 pps, not a lot isn't it?

it seems there is a lot of margin...

> The CPU level there is probably normal for that processor.
> 
> If your VIPs have enough memory you could 'ip cef distributed'.

'ip cef distributed' is not an option completed by pressing the
key '?' (only accounting option is shown, anyway i have entered it
and the output of the "show ip interfaces | include switch" is 
the same output and the ratio of high cpu interrupts has not changed.

What I find extrange is that all percentage of the cpu usage goes 
to interrupts:

Addionally (I haven't mentioned previously) is that when multicast
comes in (around 10-15 Mbps) the cpu burns at 100%. But first 
I wanted to make clear that there is no problem with unicast traffic.

Thanks so much for your comments

regards

	Ulisses

[2]

# show interfaces | include packets/sec
  5 minute input rate 62946000 bits/sec, 8537 packets/sec
  5 minute output rate 27570000 bits/sec, 7036 packets/sec
  5 minute input rate 3234000 bits/sec, 1041 packets/sec
  5 minute output rate 10653000 bits/sec, 1287 packets/sec
  5 minute input rate 23616000 bits/sec, 5363 packets/sec
  5 minute output rate 37962000 bits/sec, 5599 packets/sec
  5 minute input rate 414000 bits/sec, 82 packets/sec
  5 minute output rate 329000 bits/sec, 76 packets/sec
  5 minute input rate 594000 bits/sec, 612 packets/sec
  5 minute output rate 14166000 bits/sec, 1571 packets/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 324000 bits/sec, 76 packets/sec

> 
> 
> Rodney
> 
> On Tue, Feb 13, 2007 at 02:41:51PM +0100, ulisses wrote:
> > 
> > Hi all
> > 
> > I have followed the guidelines
> > 
> > http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml
> > http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml#high_cpu
> > 
> > trying to find why I have a high cpu usage due to interrupts
> > but I could not solve it, next I show diagnostics when only routing
> > unicast traffic.
> > 
> > The basic configuration of the system (Cisco 7500) is:
> > 
> > IOS (tm) RSP Software (RSP-JSV-M), Version 12.1(14), RELEASE SOFTWARE (fc1)
> > cisco RSP1 (R4600) processor with 65536K/2072K bytes of memory.
> > R4600 CPU at 100Mhz, Implementation 32, Rev 2.0
> > 2 AIP controllers (2 ATM).
> > 2 VIP2 controllers (3 FastEthernet).
> > 3 FastEthernet/IEEE 802.3 interface(s)
> > 2 ATM network interface(s)
> > 
> > This is the load of the router:
> > 
> > EB-LasPalmas1#show interfaces | include is up|load
> > ATM0/0 is up, line protocol is up
> >      reliability 255/255, txload 55/255, rxload 152/255
> > ATM0/0.1 is up, line protocol is up
> >      reliability 255/255, txload 55/255, rxload 152/255
> > FastEthernet1/0/0 is up, line protocol is up
> >      reliability 255/255, txload 82/255, rxload 16/255
> > FastEthernet2/0/0 is up, line protocol is up
> >      reliability 255/255, txload 119/255, rxload 68/255
> > FastEthernet3/0/0 is up, line protocol is up
> >      reliability 255/255, txload 1/255, rxload 1/255
> > FastEthernet3/1/0 is up, line protocol is up
> >      reliability 255/255, txload 35/255, rxload 1/255
> > 
> > This is the load near the peak load of the router. I also haven't 
> > seen any throttle in the interfaces. 
> > 
> > I think it is not too much, isn't it?
> > 
> > I have enabled all possible swithing options possible, see [1] at the end
> > 
> > I also have verified that there are no cache misses in the switching path
> > and traffic uses the supossed switched path with show interfaces stats.
> > 
> > The cpu load is 100% (47 of 48%) caused by interrupts:
> > 
> > EB-LasPalmas1#show process cpu | include CPU|PID|Input
> > CPU utilization for five seconds: 48%/47%; one minute: 45%; five minutes: 44%
> >  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
> >   13         176      1809         97  0.00%  0.00%  0.00%   0 ARP Input
> >   40        7036    105293         66  0.00%  0.00%  0.00%   0 ATM OAM Input
> >   44     6572080   1952147       3366  0.08%  0.09%  0.11%   0 IP Input
> >   50           0         2          0  0.00%  0.00%  0.00%   0 SSCOP Input
> >   56           0         2          0  0.00%  0.00%  0.00%   0 ILMI Input
> >   64           0         2          0  0.00%  0.00%  0.00%   0 ATMSIG Input
> >   69           0         1          0  0.00%  0.00%  0.00%   0 Probe Input
> >   70           0         1          0  0.00%  0.00%  0.00%   0 RARP Input
> >   78           0         2          0  0.00%  0.00%  0.00%   0 SPX Input
> >   83         964       926       1041  0.00%  0.00%  0.00%   0 Tag Input
> >   87           0         1          0  0.00%  0.00%  0.00%   0 Net Input
> >  100        7536     39990        188  0.00%  0.00%  0.00%   0 CLNS Input
> >  102       28504    109557        260  0.00%  0.02%  0.00%   0 IGMP Input
> > 
> > 
> > Following the guideline on high cpu usage cause interrupts I found that:
> > 
> > Possible Causes of High CPU Utilization Due to Interrupts
> >       Inappropriate Switching Path ---------- Not likely IMHO
> >       CPU Performing Alignment Corrections -- No (verified)
> >       Router Overloaded with Traffic -------- No (verified)
> >       Software Bug -------------------------- I don't know (no access to Cisco TACS)
> >       Voice Ports Configured on the Router -- No
> >       Active Asynchronous Transfer Mode (ATM) Interfaces on the Router 
> >       --------------------------------------- No (only the required ones)
> >       Many Parallel Express Forwarding (PXF) Punts to the RP 
> >       --------------------------------------  No
> > 
> > so I don't know what to check next... Any comment or suggestion?
> > 
> > Thanks so much in advance
> > 
> > 	Ulisses
> > 
> > 
> > [1] 
> > 
> > EB-LasPalmas1#show ip interface | include is up|switch
> > ATM0/0 is up, line protocol is up
> > ATM0/0.1 is up, line protocol is up
> >   IP fast switching is enabled
> >   IP fast switching on the same interface is disabled
> >   IP Flow switching is enabled
> >   IP CEF switching is enabled
> >   IP Flow switching turbo vector
> >   IP Flow CEF switching turbo vector
> >   IP multicast fast switching is enabled
> >   IP multicast distributed fast switching is disabled
> > FastEthernet1/0/0 is up, line protocol is up
> >   IP fast switching is enabled
> >   IP fast switching on the same interface is disabled
> >   IP Flow switching is enabled
> >   IP CEF switching is enabled
> >   IP Distributed switching is enabled
> >   IP Flow switching turbo vector
> >   IP Flow CEF switching turbo vector
> >   IP multicast fast switching is enabled
> >   IP multicast distributed fast switching is enabled
> > FastEthernet2/0/0 is up, line protocol is up
> >   IP fast switching is enabled
> >   IP fast switching on the same interface is disabled
> >   IP Flow switching is enabled
> >   IP CEF switching is enabled
> >   IP Distributed switching is enabled
> >   IP Flow switching turbo vector
> >   IP Flow CEF switching turbo vector
> >   IP multicast fast switching is enabled
> >   IP multicast distributed fast switching is enabled
> > FastEthernet3/0/0 is up, line protocol is up
> >   IP fast switching is enabled
> >   IP fast switching on the same interface is disabled
> >   IP Flow switching is enabled
> >   IP CEF switching is enabled
> >   IP Distributed switching is enabled
> >   IP Flow switching turbo vector
> >   IP Flow CEF switching turbo vector
> >   IP multicast fast switching is enabled
> >   IP multicast distributed fast switching is enabled
> > FastEthernet3/1/0 is up, line protocol is up
> >   IP fast switching is enabled
> >   IP fast switching on the same interface is disabled
> >   IP Flow switching is enabled
> >   IP CEF switching is enabled
> >   IP Distributed switching is enabled
> >   IP Flow switching turbo vector
> >   IP Flow CEF switching turbo vector
> >   IP multicast fast switching is enabled
> >   IP multicast distributed fast switching is enabled
> > 
> > -- 
> > _______________________________     __    ___________________________________
> >                                    /_/
> > Network Operation Center     __            __          Email:  noc at rediris.es
> > RedIRIS/Red.es              /_/  RedIRIS  /_/
> > Edificio Bronce                   __                   Tel:    + 34 912127625
> > Pza. Manuel Gómez Moreno, s/n    /_/             Urgencias:    + 34 629148201
> > E-28020 Madrid                                         Fax:    + 34 915568864
> > 
> > _________________ Spanish Academic & Research Network _______________________
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> 

-- 
_______________________________     __    ___________________________________
                                   /_/
Network Operation Center     __            __          Email:  noc at rediris.es
RedIRIS/Red.es              /_/  RedIRIS  /_/
Edificio Bronce                   __                   Tel:    + 34 912127625
Pza. Manuel Gómez Moreno, s/n    /_/             Urgencias:    + 34 629148201
E-28020 Madrid                                         Fax:    + 34 915568864

_________________ Spanish Academic & Research Network _______________________


More information about the cisco-nsp mailing list