[j-nsp] Juniper "firewall policer" inner workings

Mon Apr 11 08:32:17 EDT 2011

Keegan,
policer configuration is shown here:
http://img690.imageshack.us/img690/3162/iperftest.png

Sthaug,
thanks for explanation! So basically one might describe small buffer
vs larger buffer with following drawing:
http://img24.imageshack.us/img24/3777/leakybucket.png (I'm not very
good with Gimp :P ) Water flows our from both buckets(the 1024KB one
and the 512KB one) with consistent rate of 10Mbps. If water is poured
into the buckets with inconsistent rate, the larger buffer is less
sensitive to bursts while in case of a small buffer, high burst might
cause water to overflow(packet loss).

Chris,
with such policer:

firewall {
    policer bw-10Mbps {
        filter-specific;
        if-exceeding {
            bandwidth-limit 10m;
            burst-size-limit 512k;
        }
        then discard;
    }
}

..results between 192.168.1.1 and 192.168.2.1 were following:

IPERF:

# iperf -c 192.168.2.1 -u -fm -t60 -d -b 10m
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 0.04 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.2.1, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 0.01 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.1 port 47344 connected with 192.168.2.1 port 5001
[  3] local 192.168.1.1 port 5001 connected with 192.168.2.1 port 38624
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.0 sec  71.5 MBytes  10.0 Mbits/sec
[  4] Sent 51021 datagrams
[  3]  0.0-60.0 sec  69.9 MBytes  9.77 Mbits/sec   0.034 ms 1186/51021 (2.3%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order
[  4] Server Report:
[  4]  0.0-60.0 sec  69.9 MBytes  9.77 Mbits/sec   0.126 ms 1192/51020 (2.3%)
[  4]  0.0-60.0 sec  1 datagrams received out-of-order
#

NUTTCP:

[root@ ~]# nuttcp -4 -u -Ri10M -T1m -v 192.168.2.1
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=1024, nstream=1, port=5001 udp -> 192.168.2.1
nuttcp-t: time limit = 60.00 seconds
nuttcp-t: rate limit = 10.000 Mbps (instantaneous), 1220 pps
nuttcp-t: send window size = 9216, receive window size = 42080
nuttcp-t: 71.2344 MB in 60.12 real seconds = 1213.27 KB/sec = 9.9391 Mbps
nuttcp-t: 72951 I/O calls, msec/call = 0.84, calls/sec = 1213.39
nuttcp-t: 9.1user 50.2sys 1:00real 98% 106i+1046d 490maxrss 0+1pf 0+11723csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=1024, nstream=1, port=5001 udp
nuttcp-r: send window size = 109568, receive window size = 109568
nuttcp-r: available send window = 82176, available receive window = 82176
nuttcp-r: 69.4453 MB in 60.12 real seconds = 1182.79 KB/sec = 9.6894 Mbps
nuttcp-r: 1832 / 72944 drop/pkt 2.51% data loss
nuttcp-r: 71116 I/O calls, msec/call = 0.87, calls/sec = 1182.86
nuttcp-r: 0.0user 0.4sys 1:00real 0% 0i+0d 0maxrss 0+6pf 71045+2csw
[root@ ~]#

So basically the packet loss was pretty much the same with both
utilities. However, in order to wrap this "Juniper "firewall policer"
inner workings" topic up, the main thing is that while Juniper counts
the network layer header into the 10Mbps policer, the iperf/nuttcp
don't :)

regards,
martin

2011/4/9 Chris Tracy <ctracy at es.net>:
> Hi Martin,
>
>> Iperf UDP is indeed very bursty- I ran "iperf -c 192.168.2.1 -u -fm
>> -t60 -d -b 10m" and at the same time did "tcpdump -w iperf_dump.pcap
>> host 192.168.2.1". Then printed deltas <...>
>
> Yes, as you can see from these results, the majority of the packets are spaced very close (~11-12 microseconds apart).
>
> If you just look at the first few hundred lines of the tcpdump -ttt output, you'll also see the burst / wait / burst / wait / burst pattern.
>
>> However, if I ran "nuttcp -4 -u -Ri10M -v -T1m -w2m 192.168.2.1" and
>> at the same time did "tcpdump -w nuttcp_dump.pcap host 192.168.2.1",
>> the most popular delta values between packets are way higher than with Iperf:
>> ...
>> As you said, in case of nuttcp UDP flood, packets are much more evenly sent :)
>
> Right, and you can make nuttcp more bursty by doing -Ri10M/5 or -Ri10M/10 to instruct it to burst 5 or 10 packets at a time, respectively.  By default, nuttcp will pace packets as evenly as possible at the expense of burning a core.  Keep in mind that the behavior of iperf more closely emulates a typical application -- I don't think users would find it very acceptable to burn up their CPU just to do precise rate-limiting.  :-)
>
> So, did you lose less packets through your policer when testing with nuttcp vs iperf, assuming the same policer settings in the middle?
>
>> Could you explain this "Your interface speed appears to be GigE, so
>> you are bursting out at line rate and relying on the buffers in the
>> router to accommodate these line-rate bursts.  You should have
>> different results if you connected the host at FastE vs GigE." a bit
>> further? I mean why should I see different results with 100BASE-TX or
>> 10BASE-T?
>
> A picture is worth a thousand words....unfortunately, I have no picture to explain this, but one would sure be useful.
>
> NICs either send out packets at line-rate or not -- it's either on or off.  By instructing nuttcp/iperf to "rate-limit" to 10Mbps, you're really saying, "please don't send more than 1,250,000 bytes every second".  As you can see from the differences between iperf/nuttcp, _how_ an application handles this request depends on its implementation.  On the wire, frames are always sent at line-rate -- if the application sends a burst in a short period of time, the next device needs to have enough buffer to handle the burst as it processes and forwards.  Imagine you have a 10GigE-to-GigE speed conversion somewhere along a path, sending host is connected at 10GigE and the receiving host is connected at GigE.   Many smaller devices do not have enough buffer to handle an incoming bursty 10G flow destined to a 1G port, and you will see a lot of drops -- but nuttcp might not observe any UDP loss at all on this path because of the way it does its rate-limiting.  No policing or shaping in my example, but you're basically doing the same thing by putting a 10Mbps bottleneck in a path where frames are originating from a host connected at GigE.  I think you can see from my example -- at a high-level -- that a device receiving a burst of X bytes from a host connected at GigE has less time (requiring more buffer) to process and forward that data than if the host is connected at 100bT or 10bT (it will take 10 to 100X longer to send the same number of bytes...)
>
>> In addition, how to start nuttcp in both directions simultaneously?
>> "nuttcp -4 -u -Ri10M -v -T1m -w2m 192.168.2.1" only sends from my
>> nuttcp client to nuttcp server.
>
> There is no equivalent of iperf -d (--dualtest) in nuttcp.  You have two options:
>
> 1) start 2 servers, one on each side, then kick off a client from each side at around the same time
>
> 2) use the '-r' option to reverse the direction of the test, and kick off two tests simultaneously (only requires one server running):  nuttcp -4 -u -Ri10M -v -T1m -w2m 192.168.2.1 & nuttcp -4 -u -Ri10M -v -T1m -w2m -r 192.168.2.1
>
> You may also find it helpful to add '-i1' which will give you # of packets lost every second, so you can see whether you are losing 1pps or a bursts of 50 packets and clean in between.
>
> -Chris
>
> --
> Chris Tracy <ctracy at es.net>
> Energy Sciences Network (ESnet)
> Lawrence Berkeley National Laboratory
>
>