[c-nsp] Interesting latency spikes

Bryan bryan at tec-works.com
Sun Dec 30 22:29:37 EST 2007


John, we are seeing the latency on packets the router is forwarding, not
destined for the router.  It is as if the router is busy processing
other packets but I find no indication of the router being over loaded.



Bryan
_________________________________________________


John van Oppen wrote:
> Are you seeing spikes on forwarded packets or just on packets destined
> for the router?   Spikes on ICMP destined for the router is normal due
> to BGP scanner.   Spikes on forwarded traffic is not, can you send a
> trace route to the endpoint that shows the problem?
> 
> Thanks,
> John
> 
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Tom Storey
> Sent: Monday, December 24, 2007 2:11 AM
> To: Bryan
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Interesting latency spikes
> 
> 
> On 24/12/2007, at 7:42 PM, Bryan wrote:
> 
>> Greetings, we are running a 6500 chassis, sup2/msfc2 dual homed to 2
>> providers (yes, I know and we are planning to upgrade in the next
>> month).  We have some customers with game servers on our network and  
>> we
>> are seeing lag spikes apx every 3 minutes.  MTR shows Avg at ~12ms and
>> spikes to ~150.
>>
>> I suspect that the BGP scanner process might be causing the issues.
>> However, I never see the cpu on the router go above 80%
>>
>>          11111               4444444444
>>    1777779999977777888889999922222777771111122222444445555544
>> 100
>> 90
>> 80
>> 70
>> 60
>> 50                                 *****
>> 40                            **********
>> 30                            **********
>> 20        *****               **********
>> 10   ***********************************               *****
>>   0....5....1....1....2....2....3....3....4....4....5....5....
>>             0    5    0    5    0    5    0    5    0    5
>>
>>              CPU% per second (last 60 seconds)
>>
>>
>>    4778754688876557876468988999546888655788664678875767887657
>>    7163799555351156480612934999592332319372336057459088251731
>> 100
>> 90          **            *               *      *       *
>> 80    ***   ****   ***   *******   ***    **    ****   ***
>> 70   ****  *****   ***   *******   ***   ***    **** ******* *
>> 60   ***** ****** ***** ********* ***** ****** ************* *
>> 50  **********************************************************
>> 40  **********************************************************
>> 30  **********************************************************
>> 20  *********************###**##***#**###************#***#****
>> 10  ##########################################################
>>   0....5....1....1....2....2....3....3....4....4....5....5....
>>             0    5    0    5    0    5    0    5    0    5
>>
>>              CPU% per minute (last 60 minutes)
>>             * = maximum CPU%   # = average CPU%
>>
>> FORONA-6509-2#sh proc cpu sorted
>> CPU utilization for five seconds: 5%/2%; one minute: 13%; five  
>> minutes: 11%
>> PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
>> 263     6221468  40883926        152  2.23%  0.48%  0.41%   0 Port
>> manager per
>>  9     2723328   3661803        743  0.47%  0.66%  0.64%   0 ARP Input
>>
>> 117     2785304  12256987        227  0.39%  0.40%  0.40%   0 IP Input
>>
>> 167      477508   1339180        356  0.15%  0.05%  0.05%   0 CEF
>> process
>> 171       49200   2089338         23  0.07%  0.00%  0.00%   0 FM core
>>
>> 158      205104    175215       1170  0.07%  0.08%  0.07%   0 Adj
>> Manager
>>  6     6812124    368585      18481  0.00%  0.83%  1.15%   0 Check  
>> heaps
>>
>> <snip>
>> FORONA-6509-2#sh proc cpu sorted | include BGP
>> 114      920940   1628069        565  0.00%  0.01%  0.00%   0 BGP
>> Router
>> 115       85344    596499        143  0.00%  0.00%  0.00%   0 BGP I/O
>>
>> 121    39258684    233887     167863  0.00%  6.48%  6.36%   0 BGP  
>> Scanner
>> </snip>
>>
>> Cisco Internetwork Operating System Software
>> IOS (tm) s222_rp Software (s222_rp-ADVENTERPRISEK9_WAN-M), Version
>> 12.2(18)SXF10, RELEASE SOFTWARE (fc1)
>>
>> ROM: System Bootstrap, Version 12.2(17r)S1, RELEASE SOFTWARE (fc1)
>> BOOTLDR: s222_rp Software (s222_rp-ADVENTERPRISEK9_WAN-M), Version
>> 12.2(18)SXF10, RELEASE SOFTWARE (fc1)
>>
>> Cisco WS-C6509 (R7000) processor (revision 2.0) with 458752K/65536K
>> bytes of memory.
>> Processor board ID SCA044404GJ
>> R7000 CPU at 300Mhz, Implementation 0x27, Rev 3.3, 256KB L2, 1024KB  
>> L3 Cache
>> Last reset from power-on
>> SuperLAT software (copyright 1990 by Meridian Technology Corp).
>> X.25 software, Version 3.0.0.
>>
>> Any Ideas are greatly appreciated.
> 
> I dont believe routing is handled by the SUP, unless perhaps it cant  
> be handled by the MSFC, and the packet is punted to the SUP for  
> processing. So the spikes you are seeing probably arent related to the  
> CPU utilisation of the SUP.
> 
> Im not too familiar with the 6500/7600 series platforms, but can you  
> perhaps get CPU statistics for each of the line cards and/or the MSFC  
> and see how they are fairing up?
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/


More information about the cisco-nsp mailing list