[c-nsp] IPerf alternative

[James Bensley]

On 7 August 2017 at 08:25, CiscoNSP List <CiscoNSP_list at hotmail.com> wrote:
> Hi everyone,
>
>
> Looking for an alternative to IPerf for link testing - Having some "reliability" concerns with IPerf results (Primarily with simultaneous data transfers) - Any recommendations are greatly appreciated (10G capabilities would be fantastic, but not 100% necessary) - Has anyone used/tried Ostinato ? Primarily after a tool that can provide load testing results (simultaneous bidirectional, one direction, and UDP/TCP are primary requirements...Happy to look at a commercial product/appliance if it ticks all the boxes, but not wanting to spend a fortune :)

Hey,

That was a thread about ECMP testing a while back and various traffic
generators were discussed, it might be worth checking that out;
http://puck.nether.net/pipermail/cisco-nsp/2017-March/104692.html

If you want really high performance (above 10Gbps) Trex and other DPDK
based tools are the way forward such as
https://github.com/pktgen/Pktgen-DPDK and
https://github.com/emmericp/MoonGen.

I’ve had some good success with iPerf to be honest if I wanted to test
that QoS maps are working; I can fire up 7 instances (we use a 7 class
model) pointing at different ports on the same target server and apply
a different DSCP value to each stream and give each stream a unique
speed. For confirming that traffic is falling into the correct class
when multiple streams are present and that DSCP 0 only getting say 25%
of the link speed when its full etc, it ticks those basic boxes.

For testing (i) higher throughput that say 1Gbps or a few Gbps and/or
for testing specific traffic features and (ii) you have the time, it
is worth getting to grips with Trex/Pktgen/Moongen. I’ve only recently
found the time to do this; specifically for say testing a new switch
from a vendor, I want to spray it with every possible ethertype and
make sure none of my frames are dropped so not only does the traffic
generate need to be able to generate a range of traffic types (unlike
say iPerf which is just one “type” at a time) it also needs to be
stateful (the receiver needs to know what the sender is sending and
then record and report what was received and what wasn’t).

This can be done with some custom Lua scripts and Pktgen and/or
Moongen. I’m trying to work through this now in the lab and then make
a basic static single page web GUI so that engineers can just plug a
device in a click “go” and it will run a bunch of tests overnight and
the next day they can read the result.

In my opinion it takes a bit of time to get this all up and running
and to learn how Moongen works etc (it’s still a work in progress for
me) but I believe it will be worth it.

To go full circle though, I don’t think there is “one” app, you
probably need several apps (if you want to go the purely software
route as we are) to provide all the features of ECMP testing,
throughput testing, QoS testing, ASIC/NPU testing, and so on. As
someone said you obviously need to pay for the hardware in this case,
but a 1u server with dual 10G NIC and 32Gb RAM is £1500 (I mention the
RAM so that we can use VMs and make it a multi-purpose test box. DPDK
and VPP powered apps have something stupid like a 1% perform loss when
run then in a VM).

On a side note: Josh mentioned Etherate
(https://github.com/jwbensley/Etherate) – I wrote that as I need more
layer 2 focused testing. I had some Linux hosts and I want to test the
network between them without being strangled by the OS TCP/IP stack so
it uses raw sockets with AF_PACKET, (as Saku mentioned, you need to
use AF_PACKET in Linux if you’re not going to use a 3rd party library
like DPDK or Netmap). The main features I use for are to load a frame
from file so I can send any traffic to test an ACL filter, Ethertype
match, load-balancing pseudowires over ECMP, and these sorts of corner
cases in which you have something very specific you want to test. One
can quickly edit some hex in a text editor to generate a packet that
“triggers” or “matches” your test case and then pump that frame out
your NIC in “quick and easy” fashion like iPerf using just a few CLI
switches. This is unlike Pkgten and Moongen which have much more
overhead. Also Scappy has a friendlier and flexible method of crafting
any packet headers you desire but I’m very used to looking at hex
captures of packets so I find Etherate easy enough.

Etherate is also handy for testing things that require some bandwidth
but not loads, so it’s handy for testing STP or CoPP for example as
10Mbps of the “right” traffic can flatten a core router. With any
recent Linux Kernel you will get 1Gbps out of your laptop/desktop 1G
NIC. You can push 10Gbps using 1514 byte frames but you’ll need like a
3Ghz chip on a fully isolated core, and you’ll probably need to make a
few system teaks like lowering your NET_RX IRQ frequency, have a
recent kernel with QDISC bypass enabled, etc. It’s like saying your
car goes 100Km/h but you need to be going downhill with the wind
behind you J 64 byte packets doing 10Gbps or higher is out of the
question, go back to Pktgen/Moongen/Trex for proper high bandwidth or
high pps stuff.

I started writing a multi-threaded version of Etherate
(https://github.com/jwbensley/EtherateMT) which can get 10Gbps using
1514 byte frames with just one or two threads depending on the system.
However sending or receiving 10Gbps with 64 byte (or 84 byte) frames
is hard (you will need many cores for this). I had no intention of
adding all the stateful inspection stuff that MoonGen/PktGen/Trex
have, that would be re-inventing the wheel. Just “quick fire” high
bandwidth testing using a custom packet/hex file. I simply don’t have
the time though and that multi-threaded version if progressing
extremely slowly.

If anyone is interested in helping, I have been reading up and trying
to learn P4 (http://p4.org/) instead and the P4FPGA project
(http://p4fpga.github.io/) I believe is a better way forward (not
necessarily that exact FPGA project but something like that). Write a
stateful and flexible traffic tester that runs on an FPGA; an FPGA
with dual 10G NICs or a 40G NIC is “only” a few hundred pounds (GBP),
which is pretty cheap in the grand scheme of things (I think). There
is also progress in the P4 to XDP movement:
https://github.com/vmware/p4c-xdp Both of them interest me as
potential method of rapid development whilst still running on either
native x86 tin or FPGAs (which I consider cheap in the grand scheme of
things). Also for NFV development, where a traffic generator/testing
could be one of many open source NFVs built using P4.

Cheers,
James.