[c-nsp] Centos upload speed slower on 1000m than 100m over WAN links

[Arie Vayner (avayner)]

Paul,

Let's see if I get it strait.

If you run the server at 1G, and test locally on the same switch, you
get 100MB/s?
If you run the server at 1G, and test from the WAN (beyond the Level 3
connection) you get 1.3MB/s?
If you run the server at 100M, and test from the WAN (beyond the Level 3
connection) you get 11.5MB/s?

If you run the same test from the local 6500 (not the same switch where
the server is on, but the one used to connect to Level3) with 1G/100M?

If the port on the upstream to Level 3 is limited to a higher rate
(let's say 300M), you would be able to pass on average the full 100Mbps
rate (hence 11.5MB/s) but if you run the server at 1G, it would result
in the server being able to overload the link with bursts at 1G rate,
while the link is only (let's say) 300Mbps. This would cause Level3 to
drop packets, which would dramatically affect the TCP window state of
the download session.
This situation can be fixed by removing the rate limit or by using
egress shaping (which is not supported natively on the 6500 so it's not
easy to test).

Can you please answer the above points so we can see if we are on the
same page?
Tnx
Arie

-----Original Message-----
From: Paul [mailto:paul at gtcomm.net] 
Sent: Sunday, June 27, 2010 11:32
To: Arie Vayner (avayner)
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m over
WAN links

No, and like i said, i get 11.5MB/s transfer rate if i set my local port

to 100mbits, and i get 1.3MB/s if i set it to 1000mbps
yet internally on the same switch i get 100MB/s between servers
gigabit..
If the port was limited in any way i wouldn't get 11.5MB/s transfer rate

by setting my local port to 100mbits.


Arie Vayner (avayner) wrote:
> Paul,
>
> What kind of a link are you getting from Level3?
> Could it be a subrate link (i.e. a GigE port with some lower bandwidth
> service?)
>
> Arie
>
> -----Original Message-----
> From: Paul [mailto:paul at gtcomm.net] 
> Sent: Sunday, June 27, 2010 11:10
> To: Arie Vayner (avayner)
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m
over
> WAN links
>
> Even plugged directly into edge router (cisco 6500) connected to
level3 
> and tested on another server on level3 5 hops away.
> When the port is set at 100 i can get full 100m speed, when i set it
at 
> 1g I get less, which makes absolutely no sense and I'm totally
stumped.
>
>
> Arie Vayner (avayner) wrote:
>   
>> Paul,
>>
>> I am not really aware of the fine details on the CentOS thingie, but
>>     
> can
>   
>> you describe how the upstream network connection of that server looks
>> like? What lies beyond the NIC in the next few network hops.
>>
>> Thanks
>> Arie
>>
>> -----Original Message-----
>> From: cisco-nsp-bounces at puck.nether.net
>> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Paul
>> Sent: Sunday, June 27, 2010 09:04
>> To: cisco-nsp at puck.nether.net
>> Subject: [c-nsp] Centos upload speed slower on 1000m than 100m over
>>     
> WAN
>   
>> links
>>
>> I'm not even sure this is the right forum but since we use mainly
>>     
> Cisco 
>   
>> equipment I'll give this a shot. :)
>> I have tried several centos based servers and compiled various
kernels
>>     
>
>   
>> and the results have been extremely weird.
>> 90% of the cases the remote hosts can download from a server at 
>> 1-5megabytes per second, and most of these are over
>> the internet ranging from 30-200ms away.  Local (1ms or less) is
super
>>     
>
>   
>> fast 100MB/s for example. 
>> Ok that sounds normal since it's going over the internet, etc.  But 
>> here's the )(!@*! part..
>> If I set the port speed to 100 megabits full duplex on the switch and

>> server , the clients that get 1-5MB/s now get 11MB/s which is
>> approximately the limit of the 100mbit port. 
>> Totally stumped here, tried different nics, servers, even 4 different

>> switches.  Is a very interesting problem and I'm probing to see
>> if anyone else has encountered it. 
>> So far the only OS i have tried is centos, but different versions and

>> kernels and hardware.
>> All the switches/routers are Cisco based, but I seriously doubt that
>>     
> has
>   
>> anything to do with this. :P
>>
>>   
>>     
>
>   

-- 
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul at gtcomm.net
http://www.gtcomm.net