[c-nsp] 7609 High CPU due to IP Input and Tag Input

[Blaz Zupan]

We have three 7609 with identical hardware configurations, with two of 
them having normal CPU usage while the third one sees high CPU usage due 
to IP Input and Tag Input processes. All three machines are running 
12.2(18)SXF8 (s72033-adventerprisek9_wan-mz.122-18.SXF8.bin).

Here is what the problematic box shows:

maribor3#show proc cpu sort | ex 0.00
CPU utilization for five seconds: 24%/13%; one minute: 31%; five 
minutes: 30%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 208   102757464 739213274        139  5.27%  4.69%  4.66%   0 Tag Input
 124   2740496481519913035        180  3.67%  3.56%  3.48%   0 IP Input
   8    21967708  71492385        307  0.79%  1.04%  1.12%   0 ARP Input
   3        6272      7325        856  0.31%  0.02%  0.05%   1 SSH Process
 170     3917100    305983      12801  0.31%  0.20%  0.22%   0 Adj Manager
 176     1857240   1022835       1815  0.23%  0.09%  0.08%   0 IPC LC 
Message H
 130     2376768   7121174        333  0.15%  0.16%  0.14%   0 ARP HA
 333     2042400   7425874        275  0.07%  0.08%  0.07%   0 BGP Router
 181     1069940   2486998        430  0.07%  0.07%  0.06%   0 CEF process
 332     1001612   3902572        256  0.07%  0.03%  0.04%   0 OSPF Router 1

The ok box shows this:

ljubljana9#show proc cpu sort | ex 0.00
CPU utilization for five seconds: 9%/6%; one minute: 15%; five minutes: 14%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  45          88       105        838  1.59%  0.12%  0.02%   2 SSH Process
   8    17520028  76102308        230  0.87%  1.00%  1.03%   0 ARP Input
 124    18549000 143423921        129  0.71%  0.87%  0.82%   0 IP Input
 170     2844936    305695       9306  0.23%  0.22%  0.23%   0 Adj Manager
  86      222176    303204        732  0.07%  0.03%  0.01%   0 Compute 
load avg


The Tag Input and IP Input processes rarely go below about 3-4%, while 
the other boxes usually hover just below 1%. The "Troubleshooting High 
CPU Usage" document from Cisco did not help identifying the cause.

While I can understand the possible reasons for high CPU due to IP Input 
(traffic being process switched due to misconfiguration), I don't see 
how this could happen for Tag Input, which I guess is handling MPLS. 
What kind of misconfiguration could cause process switching of MPLS? The 
box in question runs a heavy mix of typical ISP traffic (IPv4, just a 
little bit IPv6, BGP, OSPF, MPLS L3 VPNs, multicast). The only 
difference in used protocols between the problematic box and the others 
is IPv6, but I have already tried to temporary completely shut off IPv6 
with no visible difference. The IPv6 configuration does contain tunnels, 
which are probably process switched, but the traffic is so small (almost 
nonexistant) that it shouldn't affect CPU usage at all.

Has anybody experienced this, especially the high CPU usage due to Tag 
Input process?