[c-nsp] Cisco Nexus and Port-VLAN limits

[Adam Chappell]

Hello all. Does anyone have extensive experience and familiarity with Cisco
Nexus and specifically the causes and effects of the Port-VLAN product
limit described here (mentioned as "Logical interfaces"):

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration_limits/limits_521/nexus_5000_config_limits_521.html#wp327738

The number appears to have varied in different software releases, but in
5.2.1, it's claimed as 16,000 verified, and 32,000 theoretical maximum. I
am interpreting this to mean that I can run 320 layer-2 ports if I run them
with 100 VLANs, but if I decide to run them with 1000 VLANs, I am limited
to 32 ports etc. etc (depending on the limit I trust).

I've got a topology probably not uncommon for cloud service provider
environments: multi-VLAN, MAC-aware switching for a number of 10G trunk
edge ports, which are hosting virtualisation servers with per-customer
VLAN-based logical separation.

The Nexus gives me fabulous port density, especially with the fabric
extenders, but really I want the flexibility for the edge virtualisation
servers to be able to manage the VM load and "tap into" VLANs
however/whenever they wish, without manipulating switchport trunk allow
lists, or active VLAN lists.

It isn't the case that any single trunk would have anything like 1000 VLANs
on it at any one time - but I simply want the option for the end-host to be
able to use VLANs in that space without needing switch reconfiguration.

The active topology for all VLANs would be the same, and MST is proposed
with all VLANs mapped to single instance, so there is no per-VLAN STP tax
that would cause a burden. I note though that the Cisco docs explicitly
note that these limits exist independent of STP choice, so it's something
causing the limit.

It seems though that running this sort of promiscuous trunk port
configuration seriously hampers the number of physical ports I could run.
The number isn't disastrous  but it's not ideal either, and I am very
curious to understand the causes of the limit, and possible actions I might
take to mitigate its effects.  Creative suggestions I've had so far:

- run the edge trunks as 802.1Q tunnel, so as to only really consume a
single VLAN on the Nexus.

- switch off or otherwise disable VLAN-awareness on the Nexus, or change
the EtherType recognised for VLANs.

These options are interesting, but they would both collapse the MAC
switching into a single VLAN-space as well, which I'm concerned wouldn't be
ideal for any protocols with well-known MACs (router upstreams with
VRRP/HSRP for example?).

Any experiences or suggestions welcomed and appreciated.

-- Adam.