[c-nsp] Cisco NCS VxLAN Experience

[Peter Rathlev]

On Wed, 2020-01-08 at 18:53 +0200, Alex K. wrote:
> A customer of mine's interested in acquiring some NCS boxes, in order
> to aggregate all their servers with few NCSes as possible and p2p
> connect between them (actually between few small DCs), using VxLAN.

I can't seem to find anything on NCS 5500 and VxLAN. It's mentioned a
few times in tech updates and Cisco Live presentations but nothing
more. What role would it have in a VxLAN network? Do you have more info
on that?

> On paper it looks good. NCS seems to offer good port density,
> reasonable price (as long you stick with not too complicated
> scenarios) and supports VxLAN.

For us it seemed deceptively cheap when looking at CapEx but ended up
being much more expensive when looking at recurring license costs. Make
sure you look at e.g. 3/5/7 years when calculating costs.

> As all of know, real world experience might be pretty different. Is
> anyone here use NCS for VxLAN connectivity and would like to share
> his/her thoughts and experience? That would be much appreciated.

We haven't looked at anything VxLAN related, but we did try and
shoehorn in an NCS 55A1-24H as a DC gateway router with ACI on the
south side and an existing MPLS network on the north side.

All in all it didn't work for us. My notes from the tests are below.

We specifically tested with a Bundle-Ether interface downwards for
redundancy. I think, but I haven't tested, that the limitations are the
same also for just single stand-alone physical interfaces.

- It doesn't support HSRP, despite what at least one data sheet
  says [1]. Configuration guides say nothing about HSRP and even
  though you can configure it, it doesn't work properly. So no
  seamless migration from existing HSRP gateways.

- VRRP group IDs cannot be reused between subinterfaces on the
  same bundle-ether. This seems to be an XR limitation according 
  to [2].

- You can only configure 16 different VRRD group IDs on different
  subinterfaces on the same bundle-ether. After that it complains
  in the log. Others have noted this for NCS 5504, see [3].

    LC/0/0/CPU0:Jul 15 14:33:10.759 CEST: vlan_ea[126]: %PLATFORM-DNX_VLAN_EA-3-MAX_VRRP : Only 16 VRRP sessions can be configured under a main interface BE1 (0x800002c). 
    RP/0/RP0/CPU0:Jul 15 14:33:10.765 CEST: vrrp[1195]: %IP-FHRP-3-ERR_MAX_MAC : Maximum MAC address filters for interface Bundle-Ether1 exceeded 

- You could create "l2tranport" subinterfaces and use BVI's, and
  on these you can reuse VRRP group ID. This will give you up to
  750 "subinterfaces" per physical interface, but NCS doesn't do
  Netflow on BVI's. See e.g. [4]. It's not just not supported,
  it doesn't accept the "flow" commands there.

- Regarding multicast, the platform only supports PIM-SSM (with
  SSM mapping though) and doesn't do mVPN profile 0 (default
  MDT, GRE, PIM in the core). Keep that in mind if you need to
  interface with existing things like we did. See [5].

All in all we found the NCS 5500 platform to not be suitable as a DC
gateway for us. It could probably work fine as a core P-router or in a
specific edge scenario though.

[1]: https://www.cisco.com/c/en/us/products/collateral/routers/network-convergence-system-5500-series/datasheet-c78-737935.html
[2]: https://community.cisco.com/t5/xr-os-and-platforms/asr9k-4-2-3-to-4-3-4-upgrade-vrrp-issue/td-p/2448500
[3]: https://community.cisco.com/t5/xr-os-and-platforms/ncs5504-vrrp-sessions-limit/td-p/3675139
[4]: https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5500/netflow/63x/b-netflow-cg-ncs5500-63x/b-netflow-cg-ncs5500-63x_chapter_010.html
[5]: https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5500/multicast/61x/b-ncs-5500-multicast-configuration-guide-61x/b-ncs-5500-multicast-configuration-guide-61x_chapter_01.html

-- 
Peter