[j-nsp] ex4500 best-effort drops nowhere near congested

[joel jaeggli]

On 5/2/13 10:27 AM, Jeff Wheeler wrote:
> On Wed, May 1, 2013 at 8:27 PM, ryanL <ryan.landry at gmail.com> wrote:
>> i'm guessing this is a buffer thing, but i can't explain why it only
>> happens on my 1ge ports and not when i punt the traffic over an 10ge
> Yes, it is a buffer thing.  A 10GE interface is basically never going
> to not have time to transmit frames unless it is receiving from 10 or
> more 1GE interfaces at the same instant, steadily, for long enough to
> fill the buffer; or there is at least one 10GE interface also talking
> to it.  On the other hand, two 1GE interfaces transmitting toward the
> same out-going 1GE port can fill its buffer.
>
> This is sometimes not obvious, because you look at the long-term
> traffic and see a few hundred Mb/s, thinking, "why is there packet
> loss?"  You must keep in mind that the available buffer on modern ToR
> switches is often less than 1ms worth of traffic.
ex4500 has like 4MB of shared buffer per PFE which is not really enough 
to absorb a lot of microburst type activity.
> The "buffer bloat" discussion of recent years has not done us any
> favors.  Many customers now think that buffers have historically been
> too big.  In fact, they were just often used incorrectly / configured
> badly.  Now we are not evaluating purchases based on having sufficient
> buffer, so vendors have spent years developing products that ... lack
> sufficient buffer.
Well it's a bit different circumstances between tiny CPE routers sitting 
on dsl or cable  endpoint and and a 48port 10Gbe switch. design-wise a 
TOR has either whatever shared memory fits in the asic, or a large 
external buffer. There's literally no options in between. so a 1/10Gb/s 
TOR like the force10 s60 might have 2GB of shared packet buffer, while 
an like an arista 7050s-64 would have 9MB for all the ports, assuming 
you run it as  all 10Gb/s rather than 100/1000/10000/40000 mixes of 
ports it can cut-through-forward to every port which goes a long way 
toward ameliorating your exposure to shallow buffers.