[c-nsp] nexus 5xx vpc peer keepalives

[Charles Spurgeon]

On Mon, May 03, 2010 at 10:19:53AM +1000, Lincoln Dale wrote:

> the keepalive link is not mandatory - but certainly best practice would be for it to be operational as much as possible rather than having it on a network that didn't have the same availability as your primary data path.

In our tests, unless there is a working vpc keepalive the vpc peer
link will not be brought up when an N5K reloads.

Cisco docs note that the keepalive is critical:
http://www.cisco.com/en/US/docs/switches/datacenter/sw/4_1/nx-os/interfaces/configuration/guide/if_vPC.html#wp1385808
--------------------
"The Cisco NX-OS software uses the peer-keepalive link between the vPC
peers to transmit periodic, configurable keepalive messages. You must
have Layer 3 connectivity between the peer devices to transmit these
messages; the system cannot bring up the vPC peer link unless the
peer-keepalive link is already up and running."
--------------------

We have found that if the vpc keepalive connectivity (call it "vpk"
for convenience) does not come up first, then even though the vpc peer
links are up the vpc port-channel ints on that switch will be down and
associated physical ints will be protocol down ("suspended").

Here's what you see in the log after reloading with a failing vpk:
--------------------
N5K-B %VPC-3-VPC_PEER_LINK_BRINGUP_FAILED: vPC peer-link bringup
failed (peer-keepalive not operational, peer never alive)
--------------------

If the vpk path is working at switch boot time but then fails later,
the vpc portchannels across the two switches will keep working until a
switch is reloaded, at which point the reloaded switch will fail to
find a vpc role and all vpc portchannels on the switch will stop
working.

"show vpc brief" after a switch is reloaded with a non-functional vpk
path for that vpc domain:
--------------------
vPC domain id                   : 100
Peer status                     : peer link is down
                                  (peer-keepalive not operational,
                                  peer never alive)
vPC keep-alive status           : Suspended (Destination IP not reachable)
Configuration consistency status: failed
Configuration consistency reason: Consistency Check Not Performed
vPC role                        : none established
--------------------

-Charles

Charles E. Spurgeon / UTnet
UT Austin ITS / Networking
c.spurgeon at its.utexas.edu / 512.475.9265