[c-nsp] Centos upload speed slower on 1000m than 100m over WAN links

[Paul]

Arie Vayner (avayner) wrote:
> Paul,
>
> Let's see if I get it strait.
>
> If you run the server at 1G, and test locally on the same switch, you
> get 100MB/s? 
>   
Yes and that is megabytes/second so almost gigabit speed.
> If you run the server at 1G, and test from the WAN (beyond the Level 3
> connection) you get 1.3MB/s?
>   
Yes.
> If you run the server at 100M, and test from the WAN (beyond the Level 3
> connection) you get 11.5MB/s?
>
>   
Yes
> If you run the same test from the local 6500 (not the same switch where
> the server is on, but the one used to connect to Level3) with 1G/100M?
>
>   
I plugged the server directly into the same 6500 level3 is on and made 
the incoming/outgoing routes match level3, also
changed outgoing routes to some other external neighbors we have for 
carriers just to see if it was something related to
level3 itself and it's not.  All of the ports are 1gigabit or higher for 
external peers with no rate limits.
> If the port on the upstream to Level 3 is limited to a higher rate
> (let's say 300M), you would be able to pass on average the full 100Mbps
> rate (hence 11.5MB/s) but if you run the server at 1G, it would result
> in the server being able to overload the link with bursts at 1G rate,
> while the link is only (let's say) 300Mbps. This would cause Level3 to
> drop packets, which would dramatically affect the TCP window state of
> the download session.
> This situation can be fixed by removing the rate limit or by using
> egress shaping (which is not supported natively on the 6500 so it's not
> easy to test).
>
> Can you please answer the above points so we can see if we are on the
> same page?
> Tnx
> Arie
>
> -----Original Message-----
> From: Paul [mailto:paul at gtcomm.net] 
> Sent: Sunday, June 27, 2010 11:32
> To: Arie Vayner (avayner)
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m over
> WAN links
>
> No, and like i said, i get 11.5MB/s transfer rate if i set my local port
>
> to 100mbits, and i get 1.3MB/s if i set it to 1000mbps
> yet internally on the same switch i get 100MB/s between servers
> gigabit..
> If the port was limited in any way i wouldn't get 11.5MB/s transfer rate
>
> by setting my local port to 100mbits.
>
>
> Arie Vayner (avayner) wrote:
>   
>> Paul,
>>
>> What kind of a link are you getting from Level3?
>> Could it be a subrate link (i.e. a GigE port with some lower bandwidth
>> service?)
>>
>> Arie
>>
>> -----Original Message-----
>> From: Paul [mailto:paul at gtcomm.net] 
>> Sent: Sunday, June 27, 2010 11:10
>> To: Arie Vayner (avayner)
>> Cc: cisco-nsp at puck.nether.net
>> Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m
>>     
> over
>   
>> WAN links
>>
>> Even plugged directly into edge router (cisco 6500) connected to
>>     
> level3 
>   
>> and tested on another server on level3 5 hops away.
>> When the port is set at 100 i can get full 100m speed, when i set it
>>     
> at 
>   
>> 1g I get less, which makes absolutely no sense and I'm totally
>>     
> stumped.
>   
>> Arie Vayner (avayner) wrote:
>>   
>>     
>>> Paul,
>>>
>>> I am not really aware of the fine details on the CentOS thingie, but
>>>     
>>>       
>> can
>>   
>>     
>>> you describe how the upstream network connection of that server looks
>>> like? What lies beyond the NIC in the next few network hops.
>>>
>>> Thanks
>>> Arie
>>>
>>> -----Original Message-----
>>> From: cisco-nsp-bounces at puck.nether.net
>>> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Paul
>>> Sent: Sunday, June 27, 2010 09:04
>>> To: cisco-nsp at puck.nether.net
>>> Subject: [c-nsp] Centos upload speed slower on 1000m than 100m over
>>>     
>>>       
>> WAN
>>   
>>     
>>> links
>>>
>>> I'm not even sure this is the right forum but since we use mainly
>>>     
>>>       
>> Cisco 
>>   
>>     
>>> equipment I'll give this a shot. :)
>>> I have tried several centos based servers and compiled various
>>>       
> kernels
>   
>>>     
>>>       
>>   
>>     
>>> and the results have been extremely weird.
>>> 90% of the cases the remote hosts can download from a server at 
>>> 1-5megabytes per second, and most of these are over
>>> the internet ranging from 30-200ms away.  Local (1ms or less) is
>>>       
> super
>   
>>>     
>>>       
>>   
>>     
>>> fast 100MB/s for example. 
>>> Ok that sounds normal since it's going over the internet, etc.  But 
>>> here's the )(!@*! part..
>>> If I set the port speed to 100 megabits full duplex on the switch and
>>>       
>
>   
>>> server , the clients that get 1-5MB/s now get 11MB/s which is
>>> approximately the limit of the 100mbit port. 
>>> Totally stumped here, tried different nics, servers, even 4 different
>>>       
>
>   
>>> switches.  Is a very interesting problem and I'm probing to see
>>> if anyone else has encountered it. 
>>> So far the only OS i have tried is centos, but different versions and
>>>       
>
>   
>>> kernels and hardware.
>>> All the switches/routers are Cisco based, but I seriously doubt that
>>>     
>>>       
>> has
>>   
>>     
>>> anything to do with this. :P
>>>
>>>   
>>>     
>>>       
>>   
>>     
>
>   

-- 
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul at gtcomm.net
http://www.gtcomm.net