[c-nsp] Interesting latency spikes

[Bryan]

John, we are seeing the latency on packets the router is forwarding, not
destined for the router.  It is as if the router is busy processing
other packets but I find no indication of the router being over loaded.



Bryan
_________________________________________________


John van Oppen wrote:
> Are you seeing spikes on forwarded packets or just on packets destined
> for the router?   Spikes on ICMP destined for the router is normal due
> to BGP scanner.   Spikes on forwarded traffic is not, can you send a
> trace route to the endpoint that shows the problem?
> 
> Thanks,
> John
> 
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Tom Storey
> Sent: Monday, December 24, 2007 2:11 AM
> To: Bryan
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Interesting latency spikes
> 
> 
> On 24/12/2007, at 7:42 PM, Bryan wrote:
> 
>> Greetings, we are running a 6500 chassis, sup2/msfc2 dual homed to 2
>> providers (yes, I know and we are planning to upgrade in the next
>> month).  We have some customers with game servers on our network and  
>> we
>> are seeing lag spikes apx every 3 minutes.  MTR shows Avg at ~12ms and
>> spikes to ~150.
>>
>> I suspect that the BGP scanner process might be causing the issues.
>> However, I never see the cpu on the router go above 80%
>>
>>          11111               4444444444
>>    1777779999977777888889999922222777771111122222444445555544
>> 100
>> 90
>> 80
>> 70
>> 60
>> 50                                 *****
>> 40                            **********
>> 30                            **********
>> 20        *****               **********
>> 10   ***********************************               *****
>>   0....5....1....1....2....2....3....3....4....4....5....5....
>>             0    5    0    5    0    5    0    5    0    5
>>
>>              CPU% per second (last 60 seconds)
>>
>>
>>    4778754688876557876468988999546888655788664678875767887657
>>    7163799555351156480612934999592332319372336057459088251731
>> 100
>> 90          **            *               *      *       *
>> 80    ***   ****   ***   *******   ***    **    ****   ***
>> 70   ****  *****   ***   *******   ***   ***    **** ******* *
>> 60   ***** ****** ***** ********* ***** ****** ************* *
>> 50  **********************************************************
>> 40  **********************************************************
>> 30  **********************************************************
>> 20  *********************###**##***#**###************#***#****
>> 10  ##########################################################
>>   0....5....1....1....2....2....3....3....4....4....5....5....
>>             0    5    0    5    0    5    0    5    0    5
>>
>>              CPU% per minute (last 60 minutes)
>>             * = maximum CPU%   # = average CPU%
>>
>> FORONA-6509-2#sh proc cpu sorted
>> CPU utilization for five seconds: 5%/2%; one minute: 13%; five  
>> minutes: 11%
>> PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
>> 263     6221468  40883926        152  2.23%  0.48%  0.41%   0 Port
>> manager per
>>  9     2723328   3661803        743  0.47%  0.66%  0.64%   0 ARP Input
>>
>> 117     2785304  12256987        227  0.39%  0.40%  0.40%   0 IP Input
>>
>> 167      477508   1339180        356  0.15%  0.05%  0.05%   0 CEF
>> process
>> 171       49200   2089338         23  0.07%  0.00%  0.00%   0 FM core
>>
>> 158      205104    175215       1170  0.07%  0.08%  0.07%   0 Adj
>> Manager
>>  6     6812124    368585      18481  0.00%  0.83%  1.15%   0 Check  
>> heaps
>>
>> <snip>
>> FORONA-6509-2#sh proc cpu sorted | include BGP
>> 114      920940   1628069        565  0.00%  0.01%  0.00%   0 BGP
>> Router
>> 115       85344    596499        143  0.00%  0.00%  0.00%   0 BGP I/O
>>
>> 121    39258684    233887     167863  0.00%  6.48%  6.36%   0 BGP  
>> Scanner
>> </snip>
>>
>> Cisco Internetwork Operating System Software
>> IOS (tm) s222_rp Software (s222_rp-ADVENTERPRISEK9_WAN-M), Version
>> 12.2(18)SXF10, RELEASE SOFTWARE (fc1)
>>
>> ROM: System Bootstrap, Version 12.2(17r)S1, RELEASE SOFTWARE (fc1)
>> BOOTLDR: s222_rp Software (s222_rp-ADVENTERPRISEK9_WAN-M), Version
>> 12.2(18)SXF10, RELEASE SOFTWARE (fc1)
>>
>> Cisco WS-C6509 (R7000) processor (revision 2.0) with 458752K/65536K
>> bytes of memory.
>> Processor board ID SCA044404GJ
>> R7000 CPU at 300Mhz, Implementation 0x27, Rev 3.3, 256KB L2, 1024KB  
>> L3 Cache
>> Last reset from power-on
>> SuperLAT software (copyright 1990 by Meridian Technology Corp).
>> X.25 software, Version 3.0.0.
>>
>> Any Ideas are greatly appreciated.
> 
> I dont believe routing is handled by the SUP, unless perhaps it cant  
> be handled by the MSFC, and the packet is punted to the SUP for  
> processing. So the spikes you are seeing probably arent related to the  
> CPU utilisation of the SUP.
> 
> Im not too familiar with the 6500/7600 series platforms, but can you  
> perhaps get CPU statistics for each of the line cards and/or the MSFC  
> and see how they are fairing up?
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/