[c-nsp] Interesting latency spikes

[Tom Storey]

On 24/12/2007, at 7:42 PM, Bryan wrote:

> Greetings, we are running a 6500 chassis, sup2/msfc2 dual homed to 2
> providers (yes, I know and we are planning to upgrade in the next
> month).  We have some customers with game servers on our network and  
> we
> are seeing lag spikes apx every 3 minutes.  MTR shows Avg at ~12ms and
> spikes to ~150.
>
> I suspect that the BGP scanner process might be causing the issues.
> However, I never see the cpu on the router go above 80%
>
>          11111               4444444444
>    1777779999977777888889999922222777771111122222444445555544
> 100
> 90
> 80
> 70
> 60
> 50                                 *****
> 40                            **********
> 30                            **********
> 20        *****               **********
> 10   ***********************************               *****
>   0....5....1....1....2....2....3....3....4....4....5....5....
>             0    5    0    5    0    5    0    5    0    5
>
>              CPU% per second (last 60 seconds)
>
>
>    4778754688876557876468988999546888655788664678875767887657
>    7163799555351156480612934999592332319372336057459088251731
> 100
> 90          **            *               *      *       *
> 80    ***   ****   ***   *******   ***    **    ****   ***
> 70   ****  *****   ***   *******   ***   ***    **** ******* *
> 60   ***** ****** ***** ********* ***** ****** ************* *
> 50  **********************************************************
> 40  **********************************************************
> 30  **********************************************************
> 20  *********************###**##***#**###************#***#****
> 10  ##########################################################
>   0....5....1....1....2....2....3....3....4....4....5....5....
>             0    5    0    5    0    5    0    5    0    5
>
>              CPU% per minute (last 60 minutes)
>             * = maximum CPU%   # = average CPU%
>
> FORONA-6509-2#sh proc cpu sorted
> CPU utilization for five seconds: 5%/2%; one minute: 13%; five  
> minutes: 11%
> PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
> 263     6221468  40883926        152  2.23%  0.48%  0.41%   0 Port
> manager per
>  9     2723328   3661803        743  0.47%  0.66%  0.64%   0 ARP Input
>
> 117     2785304  12256987        227  0.39%  0.40%  0.40%   0 IP Input
>
> 167      477508   1339180        356  0.15%  0.05%  0.05%   0 CEF
> process
> 171       49200   2089338         23  0.07%  0.00%  0.00%   0 FM core
>
> 158      205104    175215       1170  0.07%  0.08%  0.07%   0 Adj
> Manager
>  6     6812124    368585      18481  0.00%  0.83%  1.15%   0 Check  
> heaps
>
> <snip>
> FORONA-6509-2#sh proc cpu sorted | include BGP
> 114      920940   1628069        565  0.00%  0.01%  0.00%   0 BGP
> Router
> 115       85344    596499        143  0.00%  0.00%  0.00%   0 BGP I/O
>
> 121    39258684    233887     167863  0.00%  6.48%  6.36%   0 BGP  
> Scanner
> </snip>
>
> Cisco Internetwork Operating System Software
> IOS (tm) s222_rp Software (s222_rp-ADVENTERPRISEK9_WAN-M), Version
> 12.2(18)SXF10, RELEASE SOFTWARE (fc1)
>
> ROM: System Bootstrap, Version 12.2(17r)S1, RELEASE SOFTWARE (fc1)
> BOOTLDR: s222_rp Software (s222_rp-ADVENTERPRISEK9_WAN-M), Version
> 12.2(18)SXF10, RELEASE SOFTWARE (fc1)
>
> Cisco WS-C6509 (R7000) processor (revision 2.0) with 458752K/65536K
> bytes of memory.
> Processor board ID SCA044404GJ
> R7000 CPU at 300Mhz, Implementation 0x27, Rev 3.3, 256KB L2, 1024KB  
> L3 Cache
> Last reset from power-on
> SuperLAT software (copyright 1990 by Meridian Technology Corp).
> X.25 software, Version 3.0.0.
>
> Any Ideas are greatly appreciated.

I dont believe routing is handled by the SUP, unless perhaps it cant  
be handled by the MSFC, and the packet is punted to the SUP for  
processing. So the spikes you are seeing probably arent related to the  
CPU utilisation of the SUP.

Im not too familiar with the 6500/7600 series platforms, but can you  
perhaps get CPU statistics for each of the line cards and/or the MSFC  
and see how they are fairing up?