[c-nsp] Centos upload speed slower on 1000m than 100m over WAN links

[Paul]

Oh and to verify some things i used iperf and send big batch of udp 
traffic at 500mbit/s and it made it with 0.004% packet loss so the 
bandwidth through to the
external link isn't the problem either..

Arie Vayner (avayner) wrote:
> Paul,
>
> Let's see if I get it strait.
>
> If you run the server at 1G, and test locally on the same switch, you
> get 100MB/s?
> If you run the server at 1G, and test from the WAN (beyond the Level 3
> connection) you get 1.3MB/s?
> If you run the server at 100M, and test from the WAN (beyond the Level 3
> connection) you get 11.5MB/s?
>
> If you run the same test from the local 6500 (not the same switch where
> the server is on, but the one used to connect to Level3) with 1G/100M?
>
> If the port on the upstream to Level 3 is limited to a higher rate
> (let's say 300M), you would be able to pass on average the full 100Mbps
> rate (hence 11.5MB/s) but if you run the server at 1G, it would result
> in the server being able to overload the link with bursts at 1G rate,
> while the link is only (let's say) 300Mbps. This would cause Level3 to
> drop packets, which would dramatically affect the TCP window state of
> the download session.
> This situation can be fixed by removing the rate limit or by using
> egress shaping (which is not supported natively on the 6500 so it's not
> easy to test).
>
> Can you please answer the above points so we can see if we are on the
> same page?
> Tnx
> Arie
>
> -----Original Message-----
> From: Paul [mailto:paul at gtcomm.net] 
> Sent: Sunday, June 27, 2010 11:32
> To: Arie Vayner (avayner)
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m over
> WAN links
>
> No, and like i said, i get 11.5MB/s transfer rate if i set my local port
>
> to 100mbits, and i get 1.3MB/s if i set it to 1000mbps
> yet internally on the same switch i get 100MB/s between servers
> gigabit..
> If the port was limited in any way i wouldn't get 11.5MB/s transfer rate
>
> by setting my local port to 100mbits.
>
>
> Arie Vayner (avayner) wrote:
>   
>> Paul,
>>
>> What kind of a link are you getting from Level3?
>> Could it be a subrate link (i.e. a GigE port with some lower bandwidth
>> service?)
>>
>> Arie
>>
>> -----Original Message-----
>> From: Paul [mailto:paul at gtcomm.net] 
>> Sent: Sunday, June 27, 2010 11:10
>> To: Arie Vayner (avayner)
>> Cc: cisco-nsp at puck.nether.net
>> Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m
>>     
> over
>   
>> WAN links
>>
>> Even plugged directly into edge router (cisco 6500) connected to
>>     
> level3 
>   
>> and tested on another server on level3 5 hops away.
>> When the port is set at 100 i can get full 100m speed, when i set it
>>     
> at 
>   
>> 1g I get less, which makes absolutely no sense and I'm totally
>>     
> stumped.
>   
>> Arie Vayner (avayner) wrote:
>>   
>>     
>>> Paul,
>>>
>>> I am not really aware of the fine details on the CentOS thingie, but
>>>     
>>>       
>> can
>>   
>>     
>>> you describe how the upstream network connection of that server looks
>>> like? What lies beyond the NIC in the next few network hops.
>>>
>>> Thanks
>>> Arie
>>>
>>> -----Original Message-----
>>> From: cisco-nsp-bounces at puck.nether.net
>>> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Paul
>>> Sent: Sunday, June 27, 2010 09:04
>>> To: cisco-nsp at puck.nether.net
>>> Subject: [c-nsp] Centos upload speed slower on 1000m than 100m over
>>>     
>>>       
>> WAN
>>   
>>     
>>> links
>>>
>>> I'm not even sure this is the right forum but since we use mainly
>>>     
>>>       
>> Cisco 
>>   
>>     
>>> equipment I'll give this a shot. :)
>>> I have tried several centos based servers and compiled various
>>>       
> kernels
>   
>>>     
>>>       
>>   
>>     
>>> and the results have been extremely weird.
>>> 90% of the cases the remote hosts can download from a server at 
>>> 1-5megabytes per second, and most of these are over
>>> the internet ranging from 30-200ms away.  Local (1ms or less) is
>>>       
> super
>   
>>>     
>>>       
>>   
>>     
>>> fast 100MB/s for example. 
>>> Ok that sounds normal since it's going over the internet, etc.  But 
>>> here's the )(!@*! part..
>>> If I set the port speed to 100 megabits full duplex on the switch and
>>>       
>
>   
>>> server , the clients that get 1-5MB/s now get 11MB/s which is
>>> approximately the limit of the 100mbit port. 
>>> Totally stumped here, tried different nics, servers, even 4 different
>>>       
>
>   
>>> switches.  Is a very interesting problem and I'm probing to see
>>> if anyone else has encountered it. 
>>> So far the only OS i have tried is centos, but different versions and
>>>       
>
>   
>>> kernels and hardware.
>>> All the switches/routers are Cisco based, but I seriously doubt that
>>>     
>>>       
>> has
>>   
>>     
>>> anything to do with this. :P
>>>
>>>   
>>>     
>>>       
>>   
>>     
>
>   

-- 
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul at gtcomm.net
http://www.gtcomm.net