[c-nsp] Centos upload speed slower on 1000m than 100m over WANlinks

[Paul]

This isn't exactly the problem I am seeing.. I actually set up a windows 
server and it shows the same result
as the centos server which leads me to believe it's not a driver issue 
with centos.
None of our connections are overflowing, the transfer doesn't even start 
out fast. It's going through all gigabit or higher
ports the entire way.
One particular transfer I get 1.3MB/s every time, consistently, and if i 
disable TSO/GSO i get 8-9MB/s average but during
the transfer the rate jumps up and down a lot.    (this is on centos 
using ethtool -K eth0 tso off gso off)
Windows gets the 1.3MB/s  but i haven't tried disabling tso/gso yet.
If I set the port to 100mbits, both max it out no problem.
Locally where latency is < 1ms both come near maxing out the gigabit 
port (probably hard drive limitation)

Is there any utility that will test this end to end ?  I've used iperf 
to do loss/transfer tests. What kills me is that the servers
i'm using to test with can download at 300-400mbits from the server on 
the other end that im using to test with but can only upload
at 10mbits.   One would think, that if a server on level3 for example in 
one location and another server on level3 in another location
both on gigabit ports, should get a good rate both directions.  And of 
course if i set it to 100m, it gets 100m and not 10m..
I'm still stumped by this issue. 
Wouldn't having a server on the other end at 1gbit and using a 100m port 
to upload with cause more packet drops than having gigabit on
the uploading server?  since it maxes out the 100m port


Arie Vayner (avayner) wrote:
> Joe, this is exactly the phenomena I was referring to. It can be
> controlled with applying shaping on platforms that can support this kind
> of QOS policy (requires large buffers).
> Usually available on WAN routers, specific switches or requires specific
> modules on some other switches.
>
> Arie
>
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Joe Loiacono
> Sent: Monday, June 28, 2010 17:11
> To: Paul
> Cc: cisco-nsp-bounces at puck.nether.net; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m over
> WANlinks
>
> OK I'm jumping in on this thread late as I just got back from some 
> vacation, don't know if this particluar observation has been discussed, 
> but ...
>
> We've seen this problem a lot when moving up to new local connection 
> speeds. The problem for us has been that unless the entire path can 
> support the new speed (e.g., 1G) switches down the path that connect to 
> slower speeds (e.g. 100M) will overflow and put your data transfer into 
> TCP slow-start recovery. As soon as the sending NIC is 'downgraded'
> (e.g., 
> back to 100 M) the overflows disappear, slow-start is avoided, and 
> performance improves. Bitterly ironic.
>
> Joe
>
>
>
> From:
> Paul <paul at gtcomm.net>
> To:
> cisco-nsp at puck.nether.net
> Date:
> 06/27/2010 03:08 AM
> Subject:
> [c-nsp] Centos upload speed slower on 1000m than 100m over WAN links
>
>
>
> I'm not even sure this is the right forum but since we use mainly Cisco 
> equipment I'll give this a shot. :)
> I have tried several centos based servers and compiled various kernels 
> and the results have been extremely weird.
> 90% of the cases the remote hosts can download from a server at 
> 1-5megabytes per second, and most of these are over
> the internet ranging from 30-200ms away.  Local (1ms or less) is super 
> fast 100MB/s for example. 
> Ok that sounds normal since it's going over the internet, etc.  But 
> here's the )(!@*! part..
> If I set the port speed to 100 megabits full duplex on the switch and 
> server , the clients that get 1-5MB/s now get 11MB/s which is
> approximately the limit of the 100mbit port. 
> Totally stumped here, tried different nics, servers, even 4 different 
> switches.  Is a very interesting problem and I'm probing to see
> if anyone else has encountered it. 
> So far the only OS i have tried is centos, but different versions and 
> kernels and hardware.
> All the switches/routers are Cisco based, but I seriously doubt that has
>
> anything to do with this. :P
>
>   

-- 
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul at gtcomm.net
http://www.gtcomm.net