[f-nsp] Bigiron RX-4 weird Network Issues

[Skylar MacMinn]

Not totally sure where to start on this, so maybe someone can help.

We utilize a RX-4 with 10gbit uplinks. We have a 10gbit Atrato and a 10gbit DC blend.

Long story short, each server is connected to a router with 10gbit of connectivity at the router level, and 1gbit switch ports to the server. Each service should be able to pull on average 60M/s on a cachefly test.

Last month or so, VPS have randomly begun throttling themselves. A container currently is only able to pull 772K/s on the same speedtest a container on the same node can pull normally at 60, and the node can pull fine at 60-80. If I REBOOT the router, change the container's IP, or migrate the container to a different hardware node, the issue is fixed temporarily - eventually it starts doing it again. It happens at random, in no particular order, and we haven't been able to determine WHAT causes this to happen.


Example container with issue:
wget -O /dev/null http://cachefly.cachefly.net/100mb.test
--2014-05-21 13:50:36--  http://cachefly.cachefly.net/100mb.test
Resolving cachefly.cachefly.net... 205.234.175.175
Connecting to cachefly.cachefly.net|205.234.175.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `/dev/null'

100%[==========================================================================>] 104,857,600  772K/s   in 1m 41s

2014-05-21 13:52:17 (1016 KB/s) - `/dev/null' saved [104857600/104857600]


Node container is on:
wget -O /dev/null http://cachefly.cachefly.net/100mb.test
--2014-05-21 13:59:16--  http://cachefly.cachefly.net/100mb.test
Resolving cachefly.cachefly.net... 205.234.175.175
Connecting to cachefly.cachefly.net|205.234.175.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: "/dev/null"

100%[==========================================================================>] 104,857,600 44.3M/s   in 2.3s

2014-05-21 13:59:19 (44.3 MB/s) - "/dev/null" saved [104857600/104857600]


VPS On node with no issue:
wget -O /dev/null http://cachefly.cachefly.net/100mb.test
--2014-05-21 13:59:31--  http://cachefly.cachefly.net/100mb.test
Resolving cachefly.cachefly.net (cachefly.cachefly.net)... 205.234.175.175
Connecting to cachefly.cachefly.net (cachefly.cachefly.net)|205.234.175.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream]
Saving to: `/dev/null'

100%[==========================================================================>] 104,857,600 42.6M/s   in 2.3s

2014-05-21 13:59:33 (42.6 MB/s) - `/dev/null' saved [104857600/104857600]


Test values fluctuate, but you get the idea.

Could it be hardware, software, a configuration issue somewhere with the router? All feedback is appreciated.

At last check, DoS filtering, QoS, etc is turned off. If it isn't required we have it disabled (in an effort to remove anything as a possible issue)


Cordially Yours,

Skylar MacMinn
www.crissic.net<http://www.crissic.net>
Crissic Solutions, LLC

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/foundry-nsp/attachments/20140521/be0597af/attachment.html>