[c-nsp] High latency/jitter at times on a NPE-G1?

[Sean Watkins]

Hi,

I've got a couple of recently upgraded 7206VXRs NPE-400 -> NPE-G1s, that are
experiencing at times, great amounts of jitter/latency.

Reading over the lists for the past couple of years, there are a variety of
sources of this; we've gone ahead and opened a tac case 603658197 on it, but
that seems to be going nowhere.

We've examined the switching paths, they seem good. CEF is confirmed
running; we've begun turning off almost everything not explicity required
and still from time to time we see great amounts of jitter/latency.

Any ideas where to start looking? The only thing I see is the BGP
Router/Scanner process running every minute or so, but our latency spikes
don't correspond to its invocation?

All interfaces are running at peak, at about 60% utilization.
 

router>show proc cpu | exclude 0.00
CPU utilization for five seconds: 16%/14%; one minute: 16%; five minutes:
17%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  30      195780    134972       1450  0.15%  0.08%  0.05%   0 Net
Background   
  64     5181684     53996      95964  1.59%  1.08%  1.05%   0 IP Background

 159         856       194       4412  0.15%  0.03%  0.09%   3 Virtual Exec

router>show proc cpu | exclude 0.00
CPU utilization for five seconds: 16%/14%; one minute: 16%; five minutes:
17%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  30      195780    134974       1450  0.15%  0.08%  0.05%   0 Net
Background   
  64     5181684     53996      95964  1.59%  1.08%  1.05%   0 IP Background

 159         864       199       4341  0.15%  0.03%  0.09%   3 Virtual Exec

router>show proc cpu | exclude 0.00
CPU utilization for five seconds: 16%/14%; one minute: 16%; five minutes:
17%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  30      195780    134974       1450  0.15%  0.08%  0.05%   0 Net
Background   
  64     5181684     53996      95964  1.59%  1.08%  1.05%   0 IP Background

 159         868       204       4254  0.15%  0.03%  0.09%   3 Virtual Exec

router>show proc cpu | exclude 0.00
CPU utilization for five seconds: 16%/14%; one minute: 16%; five minutes:
17%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  30      195780    134974       1450  0.15%  0.08%  0.05%   0 Net
Background   
  64     5181684     53996      95964  1.59%  1.08%  1.05%   0 IP Background

 159         872       209       4172  0.15%  0.03%  0.09%   3 Virtual Exec

router>show proc cpu | exclude 0.00
CPU utilization for five seconds: 16%/14%; one minute: 16%; five minutes:
17%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  30      195780    134974       1450  0.15%  0.08%  0.05%   0 Net
Background   
  64     5181684     53996      95964  1.59%  1.08%  1.05%   0 IP Background

 159         880       214       4112  0.15%  0.03%  0.09%   3 Virtual Exec

router>show proc cpu | exclude 0.00
CPU utilization for five seconds: 42%/14%; one minute: 18%; five minutes:
18%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  52     1249292  18854470         66  0.23%  0.13%  0.11%   0 IP Input

 154      642284   3472343        184  0.15%  0.05%  0.05%   0 BGP Router

 157    11663628     84936     137322 27.43%  3.39%  2.36%   0 BGP Scanner

 159         884       219       4036  0.39%  0.06%  0.09%   3 Virtual Exec

router>show proc cpu | exclude 0.00
CPU utilization for five seconds: 42%/14%; one minute: 18%; five minutes:
18%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  52     1249292  18854493         66  0.23%  0.13%  0.11%   0 IP Input

 154      642284   3472345        184  0.15%  0.05%  0.05%   0 BGP Router

 157    11663628     84936     137322 27.43%  3.39%  2.36%   0 BGP Scanner

 159         888       224       3964  0.39%  0.06%  0.09%   3 Virtual Exec

router>show proc cpu | exclude 0.00
CPU utilization for five seconds: 42%/14%; one minute: 18%; five minutes:
18%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  52     1249296  18854504         66  0.23%  0.13%  0.11%   0 IP Input

 154      642284   3472345        184  0.15%  0.05%  0.05%   0 BGP Router

 157    11663628     84936     137322 27.43%  3.39%  2.36%   0 BGP Scanner

 159         896       229       3912  0.39%  0.06%  0.09%   3 Virtual Exec

router>show proc cpu | exclude 0.00
CPU utilization for five seconds: 42%/14%; one minute: 18%; five minutes:
18%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process 
  52     1249300  18854523         66  0.23%  0.13%  0.11%   0 IP Input

 154      642284   3472359        184  0.15%  0.05%  0.05%   0 BGP Router

 157    11663628     84936     137322 27.43%  3.39%  2.36%   0 BGP Scanner

 159         900       234       3846  0.39%  0.06%  0.09%   3 Virtual Exec

router>

This router is running a full table;

Sean