[j-nsp] QFX5100 buffer allocation

[Brian Rak]

On 5/16/2018 7:02 PM, Thomas Bellman wrote:
> On 2018-05-16 18:06, Brian Rak wrote:
>
>> We've been trying to track down why our 5100's are dropping traffic
>> due to lack of buffer space, even with very low link utilization.
> There's only 12 Mbyte of buffer space on the Trident II chip.  If you
> get 10 Gbit/s bursts simultaneously on two ports, contending for the
> same output port, it will only take 10 ms to fill 12 Mbyte.  (And of
> those 12 Mbyte, 3 Mbyte is used for dedicated per-port buffers, so you
> really only have ~9 Mbyte, so you would actually fill your buffers in
> 7.5-8 ms.)
>
> Do you see any actual problems due to the dropped packets?  Some people
> would have you believe that TCP suffers horribly from a single dropped
> packet, but reality is not quite that bad.  So don't chase problems
> that aren't there.
>
> Our busiest ports have drop rates at about 1 in every 15'000 packets
> (average over a few months), and so far we haven't noticed any TCP
> performance problems related to that.  (But I should note that most
> of our traffic is long-distance, to and from sites at least several
> milliseconds away from us, and often a 10-20 ms away.)
>
> That said, for Trident II / Tomahawk level of buffer sizes, I think
> it makes sense to configure them to have it all actually used, and
> not wasted on the lossless queues.
>
> You should probably also consider enabling cut-through forwarding, if
> you haven't already done so.  That should decrease the amount of buffer
> space used, leaving more available for when contention happens.
>
>
> 	/Bellman
>
A lot of what we run are game servers, which are very heavily UDP. They 
can deal with some dropped packets, but it's not ideal.

We're not even doing 10gbit of traffic, so the buffers should last at 
least a little bit.

Thanks for the tip about cut-through, we didn't have that enabled. Do 
you happen to know if it works from a 10g port to a broken out 4x10g port?

It's annoying to be dropping packets with a bunch of unused buffer space.