[c-nsp] Please help clarify bus/fabric terminology on the 6500/7600

[Nick Hilliard]

On 06/11/2009 08:11, Rick Ernst wrote:
>    - The 20Gbs fabric is used to transfer traffic directly between
> DFC-enabled line cards, bypassing the Sup.

Not quite.  All fabric enabled cards can transfer traffic directly to each 
other, as it's a crossbar fabric.  The difference between DFC and non-DFC 
enabled cards is that on a DFC enabled card, the destination path lookup is 
done locally on the card, whereas on a non DFC card, the internal path 
lookup is done by the sup720, and the line cards use the 32Gb bus as an 
out-of-band data channel for doing internal lookups.  The destination path 
lookup just tells the card which physical destination fabric path to use 
when sending the packet from one 20G fabric channel to another.

As each packet triggers a destination lookup, on a busy box pushing many 
mpps, the 32Gb bus can get saturated by lookup requests, and if this 
happens you need to use DFCs to move the lookup functionality away from the 
sup720 and into the line card.  So using a DFC will not affect switching 
speed unless you are pushing a very large number of pps.

> Other discussions I've had indicate that some combination of
> line cards can bring the whole system down to the lowest common
> denominator.

That used to be the case in certain configurations a long time ago, but not 
any more.

> Am I on track? Where does oversubscription on line cards come
> in? Is there something else I haven't covered?

Oversubscription on line cards just means that there is more edge switching 
capacity than the line card can actually handle.  So on a 6148-ge-tx card, 
the card has 48 gig ports, but in fact it can only handle 8G of traffic 
(and even then, with a strong tailwind).  On a fabric card, you have either 
1x or 2x 20G channels from the line card into the fabric.  This means that 
if you have more edge bandwidth being used than fabric capacity available, 
you can run into over subscription problems.  In practice, this tends not 
to be a problem on the 6724 / 6748 cards (whether TX or SFP), because on an 
imix system, you'll statistically only rarely run into full port saturation 
problems with every card on the box pushing line rate or near line rate. 
Oversubscription on fabric enabled cards tends to be more of a problem with 
10GE line cards for a variety of reasons - there's lots of talk about this 
in the list archives, to which I refer you.

Nick