[j-nsp] Juniper "firewall policer" inner workings

[Chris Tracy]

Hi Martin,

> Iperf UDP is indeed very bursty- I ran "iperf -c 192.168.2.1 -u -fm
> -t60 -d -b 10m" and at the same time did "tcpdump -w iperf_dump.pcap
> host 192.168.2.1". Then printed deltas <...>

Yes, as you can see from these results, the majority of the packets are spaced very close (~11-12 microseconds apart).

If you just look at the first few hundred lines of the tcpdump -ttt output, you'll also see the burst / wait / burst / wait / burst pattern.

> However, if I ran "nuttcp -4 -u -Ri10M -v -T1m -w2m 192.168.2.1" and
> at the same time did "tcpdump -w nuttcp_dump.pcap host 192.168.2.1",
> the most popular delta values between packets are way higher than with Iperf:
> ...
> As you said, in case of nuttcp UDP flood, packets are much more evenly sent :)

Right, and you can make nuttcp more bursty by doing -Ri10M/5 or -Ri10M/10 to instruct it to burst 5 or 10 packets at a time, respectively.  By default, nuttcp will pace packets as evenly as possible at the expense of burning a core.  Keep in mind that the behavior of iperf more closely emulates a typical application -- I don't think users would find it very acceptable to burn up their CPU just to do precise rate-limiting.  :-)

So, did you lose less packets through your policer when testing with nuttcp vs iperf, assuming the same policer settings in the middle?

> Could you explain this "Your interface speed appears to be GigE, so
> you are bursting out at line rate and relying on the buffers in the
> router to accommodate these line-rate bursts.  You should have
> different results if you connected the host at FastE vs GigE." a bit
> further? I mean why should I see different results with 100BASE-TX or
> 10BASE-T?

A picture is worth a thousand words....unfortunately, I have no picture to explain this, but one would sure be useful.

NICs either send out packets at line-rate or not -- it's either on or off.  By instructing nuttcp/iperf to "rate-limit" to 10Mbps, you're really saying, "please don't send more than 1,250,000 bytes every second".  As you can see from the differences between iperf/nuttcp, _how_ an application handles this request depends on its implementation.  On the wire, frames are always sent at line-rate -- if the application sends a burst in a short period of time, the next device needs to have enough buffer to handle the burst as it processes and forwards.  Imagine you have a 10GigE-to-GigE speed conversion somewhere along a path, sending host is connected at 10GigE and the receiving host is connected at GigE.   Many smaller devices do not have enough buffer to handle an incoming bursty 10G flow destined to a 1G port, and you will see a lot of drops -- but nuttcp might not observe any UDP loss at all on this path because of the way it does its rate-limiting.  No policing or shaping in my example, but you're basically doing the same thing by putting a 10Mbps bottleneck in a path where frames are originating from a host connected at GigE.  I think you can see from my example -- at a high-level -- that a device receiving a burst of X bytes from a host connected at GigE has less time (requiring more buffer) to process and forward that data than if the host is connected at 100bT or 10bT (it will take 10 to 100X longer to send the same number of bytes...)

> In addition, how to start nuttcp in both directions simultaneously?
> "nuttcp -4 -u -Ri10M -v -T1m -w2m 192.168.2.1" only sends from my
> nuttcp client to nuttcp server.

There is no equivalent of iperf -d (--dualtest) in nuttcp.  You have two options:

1) start 2 servers, one on each side, then kick off a client from each side at around the same time

2) use the '-r' option to reverse the direction of the test, and kick off two tests simultaneously (only requires one server running):  nuttcp -4 -u -Ri10M -v -T1m -w2m 192.168.2.1 & nuttcp -4 -u -Ri10M -v -T1m -w2m -r 192.168.2.1

You may also find it helpful to add '-i1' which will give you # of packets lost every second, so you can see whether you are losing 1pps or a bursts of 50 packets and clean in between.

-Chris

--
Chris Tracy <ctracy at es.net>
Energy Sciences Network (ESnet)
Lawrence Berkeley National Laboratory