[c-nsp] iSCSI, port buffers, and small switches

[John Gill]

Hi Nick,
I agree they cannot be compared easily, and the reason I thought of the 
aside was to point out this kind of discrepancy, in case one were 
tempted to just use simple comparison of buffer sizes.  Ingress vs. 
egress buffering changes things significantly.

I hadn't considered your angle, let me think out loud:

If a 3750 were to have no congestion, then the buffers would be 
routinely populated with one frame, or less, per active source interface 
because it's store and forward.  If the n5k were to have no congestion, 
we only store enough to forward the frame and pass QoS/ACL checks, so 
yes it's less usage during no congestion.  As soon as either platform 
sees congestion, we behave in a store-and-forward manner.

I think there is a difference in having that buffer potentially less 
utilized, but it's going to be the space of a single frame at best in 
the 3750 and always just enough to make fwding decisions in the N5k, so 
it is true you will have more available buffer before congestion starts. 
  However, once congestion starts, that stops being true, so I guess 
there is an advantage to having that buffer available at the start of 
the congestion.  Depending on your MTU, this can buy you potentially 
around like 9k per interface - add on that the 3750 has some buffers 
shared and some reserved compared to the N5k using dedicated buffers per 
source, things look quite different in the datacenter than in the 
whitepapers, indeed.

Regards,
John Gill
cisco


On 7/27/11 4:37 AM, Nick Hilliard wrote:
> On 27/07/2011 05:00, John Gill wrote:
>> This paper is indeed a great read, it goes over some really good
>> points of
>> the buffering and queuing that are commonly misunderstood.
>>
>> All good points in this thread, I will make one small comment about
>> the N5k
>> here if one wanted to compare buffer sizes on paper: The N5k uses ingress
>> buffering with virtual output queues. So when you oversubscribe a single
>> egress interface, buffers available for use are proportional to the
>> number
>> sending to that interface. It essentially acts like a shared buffer.
>
> thing is, you can't really do a paper comparison of the n5k and
> 3560/3750 buffers - one model is store-n-forward, while the other is a
> cut-thru. This means that even though the n5k has relatively modest
> buffers by comparison, they go much further during normal operation
> because they're not routinely used unless there is port congestion.
> Unless of course, you mix-n-match 1G and 10G on the same chassis, which
> causes the n5k to implement per port store-n-forward on the 1G <-> 10G
> paths. However, this is usually a very poor idea (unless you know what
> you're doing).
>
> Interestingly, the Miercom-supervised Nexus5k vs Arista 7124 lab test of
> April 2010 uses ingress buffering + VOQs on the N5K to concoct some of
> their more extraordinary (but fully repeatable) results. I love that
> paper: the comparison methodology between the two switches is completely
> hilarious.
>
> Nick
>