[c-nsp] Centos upload speed slower on 1000m than 100m over WANlinks

[Arie Vayner (avayner)]

Paul,

The bursts are sub-second, and you would not see them on the transfer
rate over time or using 30-seconds average counters on the switches (you
would see them with a sniffer...)

I suggest you test this in a structured way, so that we know where the
problem is coming from, and then we know which part of the solution to
fix.

In order to test, you should repeat the download test with 100M and 1G
settings at each of the following locations:
1. Same access switch
2. On the 6500 before the WAN connection
3. From Level 3 (beyond the WAN connection)

The locations above are for the client ; the server should remain at the
same place.
Each location should be tested twice: With the server set to 100M and to
1G.

Another thing is to validate that the WAN link is capable of providing a
full 1G rate. If you can run an iperf test over it at 1G (outgoing!) it
would allow us to validate this part.

Thanks
Arie

-----Original Message-----
From: Paul [mailto:paul at gtcomm.net] 
Sent: Monday, June 28, 2010 18:58
To: Arie Vayner (avayner)
Cc: Joe Loiacono; cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m over
WANlinks

This isn't exactly the problem I am seeing.. I actually set up a windows

server and it shows the same result
as the centos server which leads me to believe it's not a driver issue 
with centos.
None of our connections are overflowing, the transfer doesn't even start

out fast. It's going through all gigabit or higher
ports the entire way.
One particular transfer I get 1.3MB/s every time, consistently, and if i

disable TSO/GSO i get 8-9MB/s average but during
the transfer the rate jumps up and down a lot.    (this is on centos 
using ethtool -K eth0 tso off gso off)
Windows gets the 1.3MB/s  but i haven't tried disabling tso/gso yet.
If I set the port to 100mbits, both max it out no problem.
Locally where latency is < 1ms both come near maxing out the gigabit 
port (probably hard drive limitation)

Is there any utility that will test this end to end ?  I've used iperf 
to do loss/transfer tests. What kills me is that the servers
i'm using to test with can download at 300-400mbits from the server on 
the other end that im using to test with but can only upload
at 10mbits.   One would think, that if a server on level3 for example in

one location and another server on level3 in another location
both on gigabit ports, should get a good rate both directions.  And of 
course if i set it to 100m, it gets 100m and not 10m..
I'm still stumped by this issue. 
Wouldn't having a server on the other end at 1gbit and using a 100m port

to upload with cause more packet drops than having gigabit on
the uploading server?  since it maxes out the 100m port


Arie Vayner (avayner) wrote:
> Joe, this is exactly the phenomena I was referring to. It can be
> controlled with applying shaping on platforms that can support this
kind
> of QOS policy (requires large buffers).
> Usually available on WAN routers, specific switches or requires
specific
> modules on some other switches.
>
> Arie
>
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Joe Loiacono
> Sent: Monday, June 28, 2010 17:11
> To: Paul
> Cc: cisco-nsp-bounces at puck.nether.net; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m
over
> WANlinks
>
> OK I'm jumping in on this thread late as I just got back from some 
> vacation, don't know if this particluar observation has been
discussed, 
> but ...
>
> We've seen this problem a lot when moving up to new local connection 
> speeds. The problem for us has been that unless the entire path can 
> support the new speed (e.g., 1G) switches down the path that connect
to 
> slower speeds (e.g. 100M) will overflow and put your data transfer
into 
> TCP slow-start recovery. As soon as the sending NIC is 'downgraded'
> (e.g., 
> back to 100 M) the overflows disappear, slow-start is avoided, and 
> performance improves. Bitterly ironic.
>
> Joe
>
>
>
> From:
> Paul <paul at gtcomm.net>
> To:
> cisco-nsp at puck.nether.net
> Date:
> 06/27/2010 03:08 AM
> Subject:
> [c-nsp] Centos upload speed slower on 1000m than 100m over WAN links
>
>
>
> I'm not even sure this is the right forum but since we use mainly
Cisco 
> equipment I'll give this a shot. :)
> I have tried several centos based servers and compiled various kernels

> and the results have been extremely weird.
> 90% of the cases the remote hosts can download from a server at 
> 1-5megabytes per second, and most of these are over
> the internet ranging from 30-200ms away.  Local (1ms or less) is super

> fast 100MB/s for example. 
> Ok that sounds normal since it's going over the internet, etc.  But 
> here's the )(!@*! part..
> If I set the port speed to 100 megabits full duplex on the switch and 
> server , the clients that get 1-5MB/s now get 11MB/s which is
> approximately the limit of the 100mbit port. 
> Totally stumped here, tried different nics, servers, even 4 different 
> switches.  Is a very interesting problem and I'm probing to see
> if anyone else has encountered it. 
> So far the only OS i have tried is centos, but different versions and 
> kernels and hardware.
> All the switches/routers are Cisco based, but I seriously doubt that
has
>
> anything to do with this. :P
>
>   

-- 
GloboTech Communications
Phone: 1-514-907-0050 x 215
Toll Free: 1-(888)-GTCOMM1
Fax: 1-(514)-907-0750
paul at gtcomm.net
http://www.gtcomm.net