[c-nsp] Nexus vPC loop avoidance details?

[Adrian Chung]

We've got 2 6500s (.145 and .146) connected to 2 Nexus 7Ks (.148, .149)
running 5.1.3.

The 6500s each have two ten gigE interfaces in a port-channel connected up
to vPCs on the 7K side.  On top of this, each 6500 is forming an OSPF
adjacency with each 7K.  The adjacencies form without a problem, and we're
not using peer-gateway.

I know this isn't a supported configuration, and we're looking at changing
it, but in testing I've seen some weird scenarios that I can partly
explain based on vPC loop avoidance, but other scenarios which I think
should be blocked but don't seem to be.

Here are a few scenarios, all based on two destination IPs .1 (an SVI on
the 7K) and .10 (a server) in a subnet that is learned via OSPF and
preferred via .149 (7K-2).  The destination subnet is routed from the 7Ks
over a vPC to a service chassis on a transit subnet which goes to an FWSM.
 The inside interface of the FWSM is on another transit subnet that comes
back up the same vPC to a different VRF on the 7Ks which the destination
subnet resides in.

1) from 6500 #1 (.145) attempting to ping an address ending in .10.  This
works, with no packet loss.  If I calculate the port-channel hash on
6500-1 it uses the port-channel member that goes to 7K-2 (which is also
where the L3 nexthop is.  7K-2 from what I can gather should send the
traffic out its local vPC member port facing the service chassis.  On the
service chassis, return packets hash to the port-channel member which goes
to 7K-2.  No cross vPC peer-link traffic.

2) from 6500-1 (.145) attempting to ping the .1 IP which is an HSRP IP on
an SVI on the 7Ks.  This doesn't work at all.  Port-channel hash on 6500-1
sends to port-channel member that goes to 7K-1.  Since the nexthop is .149
and is across the vPC peer-link, my understanding is it gets the loop
avoidance bit set and sent across the peer-link, but since the egress port
to the service chassis is on a vPC, and the vPC member port on 7K-1 is
active, the packet gets dropped.

That makes sense so far.  From 6500-2 (.146) the opposite scenarios work,
can ping .1, but not .10.  For both, behaviour is consistent in that
anything that crosses the peer-link gets filtered/dropped...

But when testing with an IP that is upstream from the 6500s (204.x.x.x) I
can ping both the .1 and .10 IPs with absolutely no problems, even when
the traffic appears to cross the peer-link and takes the same path as 1)
and 2) above.

I'm trying to figure out what I'm missing or misunderstanding, it seems
like all scenarios should fail equally as long as the vPC peer-link is
involved in the data path, but this doesn't seem to be case.