[c-nsp] Please help clarify bus/fabric terminology on the 6500/7600

[Geoffrey Pendery]

Well you're off to an excellent start.  Others have added some good
clarifications and details, but so far I don't see this one answered:

"Other discussions I've had indicate that some combination of line
cards can bring the whole system down to the lowest common
denominator."

My guess is that this is referring to the Fabric/Bus mode, for the chassis.
It's described on the link you sent, if you search to "Cisco Catalyst
6500 Architecture: Bus Switching Modes"

As Nick Hilliard explained, the bus is used, even with all fabric
cards, for communication between the Sup and the line cards.  The Sup
first determines which of the three modes to use for communication.

If you have a Sup with no fabric (like Sup 1A, or Sup 2 w/o SFM, or
Sup 32) the switch will run in "Flow-Through" mode, meaning that each
time a packet is received, the entire packet is sent on the shared
bus, so it's seen by the Sup and all line cards.  This will only get
you up to 15 Mpps, and a theoretical max of 32 Gbps (likely lower in
practice).

If you have a fabric Sup and fabric line cards, but at least one
Classic line card, the switch will drop into "Truncated" mode.  This
is likely what someone was referring to when they told you "lowest
common denominator".  The classic cards will still send the whole
packet, like in flow-through mode, but the fabric cards will send only
the headers, and send the data portion to the Sup via fabric.  This is
still limited to 15 Mpps, but because the data flows via fabric, you
can squeeze some extra bandwidth out.

Lastly, if you have no Classic cards present in the chassis, it can go
into Compact mode, where only compressed headers are sent via the bus,
all data flows via fabric.  This gets you up to 30 Mpps and your
theoretical 720 Gbps of total forwarding capacity.

Here's some sample output from a chassis with all fabric (and in this
case, dCEF) cards:

hostname#show fabric
show fabric active:
Active fabric card in slot 5
No backup fabric card in the system
show fabric mode:
Global switching mode is Compact
dCEF mode is not enforced for system to operate
Fabric module is not  required for system to operate
Modules are allowed to operate in bus mode
Truncated mode is allowed, due to presence of DFC module
Module Slot     Switching Mode
    1                     dCEF
    2                     dCEF
    5                     dCEF
    9                     dCEF


-Geoff


On Fri, Nov 6, 2009 at 1:11 AM, Rick Ernst <cnsp at shreddedmail.com> wrote:
>
> I'm trying to wrap my brain around Cisco's document on the 6500/7600
> technology:
>
> http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html
>
> Terminology on the bus architecture and switch fabric are becoming less
> confusing to me the more I read it, but I'm still not comfortable with my
> level of understanding.
> What I think the document says is:
>
>  - The 32Gbs shared bus is the path between the supervisor and individual
> line cards. Line cards do not move data between each other; traffic must
> pass through the Sup.
>  - The raw capacity of the 32Gbs bus is just that; 32Gbs across the entire
> bus, combined across all cards
>  - The switch fabric is single or dual channel 20Gbs, dual channel just
> allowing higher port/speed-density on the line cards
>  - The 20Gbs fabric is used to transfer traffic directly between
> DFC-enabled line cards, bypassing the Sup.
>  - The 20Gbs fabric is not shared, each DFC line card can talk to any other
> DFC line card at 20Gbs up to a potential aggregate of 720Gbs
>  - CEF and dCEF simply refer to whether the line card has a DFC
>  - CEF256 using 8Gbs of the fabric, CEF720 uses 20Gbs
>  - "Classic" line cards use only the 32Gbs bus.
>  - Usage of 8Gbs or 20Gbs on the fabric is dependent on the line card and
> the Sup.
>  - Sup720 allows 20Gbs, others are only 8Gbs
>  - Mixed 8/20Gbs line cards can be used. 20Gbs is not lost if 8Gbs is
> present.
>  - The full "720Gbs" capacity is all dual-fabric line cards with DFCs
>
> I'm most confused on the 8Gbs limit and how it relates to the Supervisor and
> line cards.  Other discussions I've had indicate that some combination of
> line cards can bring the whole system down to the lowest common
> denominator.  Am I on track? Where does oversubscription on line cards come
> in? Is there something else I haven't covered?
>
> Sorry for the laundry list.  I'd rather make sure I'm clear in my head
> before the design, then find a gotcha after it is too late.
>
> Thanks!
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/