[c-nsp] Wierd MPLS/VPLS issue

[Simon Lockhart]

On Fri Nov 04, 2016 at 03:40:05PM +0000, Simon Lockhart wrote:
> To me, everything *looks* right, it's just that some VPLS traffic traversing
> the new link gets lost.

For those who are interested...

Well, I finally got to the bottom of this, and have pushed it to Cisco TAC
for a fix...

This packet gets forwarded:

  Frame 1: 140 bytes on wire (1120 bits), 140 bytes captured (1120 bits)
  Ethernet II, Src: (00:1f:9e:08:a5:c0), Dst: (00:1a:30:0d:c8:00)
  802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 101
  MultiProtocol Label Switching Header, Label: 95, Exp: 0, S: 0, TTL: 254
  MultiProtocol Label Switching Header, Label: 1240, Exp: 0, S: 1, TTL: 4
  Ethernet II, Src: (ec:c8:82:d1:aa:ce), Dst: (88:f0:31:55:8a:50)
  Internet Protocol Version 4, Src: 5.151.211.131, Dst: 5.151.211.130
  Internet Control Message Protocol

This packet doesn't:

  Frame 1: 140 bytes on wire (1120 bits), 140 bytes captured (1120 bits)
  Ethernet II, Src: (00:1f:9e:08:a5:c0), Dst: (00:1a:30:0d:c8:00)
  802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 101
  MultiProtocol Label Switching Header, Label: 95, Exp: 0, S: 0, TTL: 254
  MultiProtocol Label Switching Header, Label: 1240, Exp: 0, S: 1, TTL: 4
  Ethernet II, Src: (ec:c8:82:d1:aa:ce), Dst: (4c:4e:35:d6:e4:50)
  Internet Protocol Version 4, Src: 5.151.211.131, Dst: 5.151.211.129
  Internet Control Message Protocol

The important difference is the Dst MAC address in the inner Ethernet II 
header. If the first nibble of the Dst MAC address is 4 or 6, the packet 
doesn't get forwarded. If it starts with anything else, it does get forwarded.

It looks like the Nexus 92160YC-X is spotting the 4 or 6 there, assuming it's
an IPv4 or IPv6 header next (Wireshark makes exactly the same incorrect 
assumption!), trying to decode it, and failing (because it's actually an
Ethernet II header), and then fails to forward the packet.

I can only assume the Nexus is looking this deep in the packet to get some
entropy for load-balancing hashing.

Trying to persuade Cisco TAC that this is a real problem with the Nexus, and
not a problem with the packet has been a real challenge. Fingers crossed I've
finally persuaded them to accept that it's their problem.

Simon
-- 
Simon Lockhart |   * Server Co-location * ADSL * Domain Registration *
   Director    |  * Domain & Web Hosting * Connectivity * Consultancy * 
  Bogons Ltd   | *  http://www.bogons.net/  *  Email: info at bogons.net  *