[c-nsp] Interesting latency spikes

[Bryan]

Greetings, we are running a 6500 chassis, sup2/msfc2 dual homed to 2
providers (yes, I know and we are planning to upgrade in the next
month).  We have some customers with game servers on our network and we
are seeing lag spikes apx every 3 minutes.  MTR shows Avg at ~12ms and
spikes to ~150.

I suspect that the BGP scanner process might be causing the issues.
However, I never see the cpu on the router go above 80%

           11111               4444444444
     1777779999977777888889999922222777771111122222444445555544
100
 90
 80
 70
 60
 50                                 *****
 40                            **********
 30                            **********
 20        *****               **********
 10   ***********************************               *****
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5

               CPU% per second (last 60 seconds)


     4778754688876557876468988999546888655788664678875767887657
     7163799555351156480612934999592332319372336057459088251731
100
 90          **            *               *      *       *
 80    ***   ****   ***   *******   ***    **    ****   ***
 70   ****  *****   ***   *******   ***   ***    **** ******* *
 60   ***** ****** ***** ********* ***** ****** ************* *
 50  **********************************************************
 40  **********************************************************
 30  **********************************************************
 20  *********************###**##***#**###************#***#****
 10  ##########################################################
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5

               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

FORONA-6509-2#sh proc cpu sorted
CPU utilization for five seconds: 5%/2%; one minute: 13%; five minutes: 11%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 263     6221468  40883926        152  2.23%  0.48%  0.41%   0 Port
manager per
   9     2723328   3661803        743  0.47%  0.66%  0.64%   0 ARP Input

 117     2785304  12256987        227  0.39%  0.40%  0.40%   0 IP Input

 167      477508   1339180        356  0.15%  0.05%  0.05%   0 CEF
process
 171       49200   2089338         23  0.07%  0.00%  0.00%   0 FM core

 158      205104    175215       1170  0.07%  0.08%  0.07%   0 Adj
Manager
   6     6812124    368585      18481  0.00%  0.83%  1.15%   0 Check heaps

<snip>
FORONA-6509-2#sh proc cpu sorted | include BGP
 114      920940   1628069        565  0.00%  0.01%  0.00%   0 BGP
Router
 115       85344    596499        143  0.00%  0.00%  0.00%   0 BGP I/O

 121    39258684    233887     167863  0.00%  6.48%  6.36%   0 BGP Scanner
</snip>

Cisco Internetwork Operating System Software
IOS (tm) s222_rp Software (s222_rp-ADVENTERPRISEK9_WAN-M), Version
12.2(18)SXF10, RELEASE SOFTWARE (fc1)

ROM: System Bootstrap, Version 12.2(17r)S1, RELEASE SOFTWARE (fc1)
BOOTLDR: s222_rp Software (s222_rp-ADVENTERPRISEK9_WAN-M), Version
12.2(18)SXF10, RELEASE SOFTWARE (fc1)

Cisco WS-C6509 (R7000) processor (revision 2.0) with 458752K/65536K
bytes of memory.
Processor board ID SCA044404GJ
R7000 CPU at 300Mhz, Implementation 0x27, Rev 3.3, 256KB L2, 1024KB L3 Cache
Last reset from power-on
SuperLAT software (copyright 1990 by Meridian Technology Corp).
X.25 software, Version 3.0.0.

Any Ideas are greatly appreciated.

-- 
_________________________
Bryan Welch
Tec-Works Inc
 Total Network Solutions
_________________________