[j-nsp] QFX5100 buffer allocation

[Thomas Bellman]

On 2018-05-16 18:06, Brian Rak wrote:

> We've been trying to track down why our 5100's are dropping traffic
> due to lack of buffer space, even with very low link utilization.

There's only 12 Mbyte of buffer space on the Trident II chip.  If you
get 10 Gbit/s bursts simultaneously on two ports, contending for the
same output port, it will only take 10 ms to fill 12 Mbyte.  (And of
those 12 Mbyte, 3 Mbyte is used for dedicated per-port buffers, so you
really only have ~9 Mbyte, so you would actually fill your buffers in
7.5-8 ms.)

Do you see any actual problems due to the dropped packets?  Some people
would have you believe that TCP suffers horribly from a single dropped
packet, but reality is not quite that bad.  So don't chase problems
that aren't there.

Our busiest ports have drop rates at about 1 in every 15'000 packets
(average over a few months), and so far we haven't noticed any TCP
performance problems related to that.  (But I should note that most
of our traffic is long-distance, to and from sites at least several
milliseconds away from us, and often a 10-20 ms away.)

That said, for Trident II / Tomahawk level of buffer sizes, I think
it makes sense to configure them to have it all actually used, and
not wasted on the lossless queues.

You should probably also consider enabling cut-through forwarding, if
you haven't already done so.  That should decrease the amount of buffer
space used, leaving more available for when contention happens.


	/Bellman

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL: <https://puck.nether.net/pipermail/juniper-nsp/attachments/20180517/4e41cb22/attachment.sig>