[c-nsp] VSS/12.2(33)SXI2a High interrupt load on SP

[Daniska, Tomas]

Hi,

 

wonder if anyone came to this... 

 

XXXXXXXXXX-sp#sh proc cpu

CPU utilization for five seconds: 100%/88%; one minute: 100%; five
minutes: 100%

 

and lasts for a week or two already. It's in ios-base, TID 6

 

XXXXXXXXXX-sp#sh proc cpu det 12311

CPU utilization for five seconds: 99%/82%; one minute: 100%; five
minutes: 100%

PID/TID   5Sec    1Min     5Min Process             Prio  STATE
CPU

12311    96.2%   97.2%    97.0% ios-base
37d06h

          0.0%    1.9%     2.8% [dead threads]

      1   0.0%    0.6%     0.7%                       10  Receive
45.837

      2   0.1%    0.0%     0.0%                        5  Ready
36m47s

      4   0.0%    0.3%     0.8%                       10  Receive
8.353

      5   0.0%    0.0%     0.0%                       11  Nanosleep
5m54s

      6  77.7%   84.2%    84.7%                       21  Intr
20d22h

 

 

I cannot come to any source of this, no log messages indicating an event
that would cause this.

 

The only thing I have noticed is that IBC port on the SP is seeing
unusually high number of inband interrupts:

 

XXXXXXXXXX-sp#sh clock

14:31:51.917 GMT Tue Dec 15 2009

XXXXXXXXXX-sp#sh ibc | i inter

        3699589169 inband interrupts

        1194856 total tx interrupts set

        mistral tx interrupt inconsisteny occured 0 times

        0 total packets dropped on throttled interfaces (0 low, 0
medium, 0 high)

 

XXXXXXXXXX-sp#sh clock        

14:32:04.201 GMT Tue Dec 15 2009

XXXXXXXXXX-sp#sh ibc | i inter

        3700120842 inband interrupts

        1195039 total tx interrupts set

        mistral tx interrupt inconsisteny occured 0 times

        0 total packets dropped on throttled interfaces (0 low, 0
medium, 0 high)

XXXXXXXXXX-sp#   

 

 

which gives >40k interrupts per seconds constantly - no wonder the SP
CPU is that busy. This, however, does not seem being related to IBC
traffic:

 

XXXXXXXXXX-sp#sh ibc load 30

Interface information:

        Interface IBC0/0/0(idb 0x60011A70)

        Hardware is Mistral IBC (revision 5)

        0 minute rx rate 359000 bits/sec, 213 packets/sec

        0 minute tx rate 490000 bits/sec, 466 packets/sec

 

 

and the RP does not seem sending much traffic over IBC as well:

XXXXXXXXXX#sh ibc load 30

Interface information:

        Interface IBC0/0(idb 0x60011A70)

        Hardware is Mistral IBC (revision 5)

        0 minute rx rate 2276000 bits/sec, 523 packets/sec

        0 minute tx rate 100000 bits/sec, 137 packets/sec

 

 

Does anyone have any hints before I proceed to TAC? Getting the SR
queued is complicated for me with this customer as they have support
bought via a different organization and it takes three human hops to
open a ticket. Through India... aww

 

thx!

 

 

--

Tomas Daniska
systems engineer

Soitron, a.s.
Plynarenska 5, 829 75 Bratislava, Slovakia
tel: +421 2 58224111, fax: +421 2 58224199

All generalizations are false, including this one.
-- Mark Twain