[c-nsp] Nexus 5xxx VPC peer keepalives

[Ryan West]

Charles,

> -----Original Message-----
> Sent: Wednesday, April 28, 2010 1:36 PM
> To: nsp-cisco
> Subject: [c-nsp] Nexus 5xxx VPC peer keepalives
> 
> Anyone,
> 
> 	Coming up on a design issue with our upcoming first deployment of Nexus
> 5010s and 5020s in a new datacenter.   It's recommended in the following doc
> to use the mgmt0 interface for peer keepalive messages:
> 
> http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/Cisco_
> Nexus_5000_Series_NX-
> OS__chapter8.html#concept_47F7274E5FDA489884D0488BC491B066
> 
> We're doing a true out of band management approach on this new network, so the
> mgmt0 interfaces all home back to an OOB switch/router (4507)  which houses
> the NMS gear, etc.  My concern is that a reload (or failure of some type) on
> this OOB switch could cause a 'dual active' situation on all the Nexus pairs
> of devices .  (6 pairs of 5010s, and the pair of 5020s that aggregate the 5010
> pairs).  I don't think I want that to happen.  So the alternative seems to be
> a back to back non-VPC-peer link between the two devices using a VLAN
> interface, but I hate the idea of using a 10 gig port just for keepalives.
> There are what appears to be additional copper mgmt ports on the boxes, but
> they're covered up, and not in the CLI.  Any way to utilize those?  Any other
> possibilities I'm overlooking?  Or am I stuck getting 1 gig copper SFPs and
> crossover cables for keepalives?
> 

If the peer-link is lost, the secondary vPC shuts its ports down.  If the peer-keepalive link is lost, the vPC remains up as all the traffic is still passing through the vPC peer link.  I didn't find any great documentation that states that, but I forced management down through 4507R port as well and it's still up.  

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572835-00_NX-OS_vPC_DG.pdf

You'll only end up with a dual-active if both the peer-link and peer-keepalive link are disconnected.  Cisco claims the worst that will happen is duplicate frames.


show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 100
Peer status                     : peer adjacency formed ok
vPC keep-alive status           : peer is not alive
Configuration consistency status: success
vPC role                        : primary

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans
--   ----   ------ --------------------------------------------------
1    Po20   up     1,400

vPC status
----------------------------------------------------------------------------
id     Port        Status Consistency Reason                     Active vlans
------ ----------- ------ ----------- -------------------------- -----------
1      Po1         up     success     success                    400

show port-channel summary
Flags:  D - Down        P - Up in port-channel (members)
        I - Individual  H - Hot-standby (LACP only)
        s - Suspended   r - Module-removed
        S - Switched    R - Routed
        U - Up (port-channel)
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SU)     Eth      LACP      Eth1/1(P)    Eth1/2(P)    Eth1/3(P)
                                     Eth1/4(P)

And the other end of a vPC port channel:

21     Po21(SU)        LACP      Gi1/0/21(P) Gi1/0/22(P) Gi1/0/23(P)
                                 Gi1/0/24(P) Gi2/0/21(P) Gi2/0/22(P)
                                 Gi2/0/23(P) Gi2/0/24(P)

So, it's still replicating the LACP ID properly. 

Channel group 21 neighbors

Partner's information:

                  LACP port                        Oper    Port     Port
Port      Flags   Priority  Dev ID          Age    Key     Number   State
Gi1/0/21  SA      32768     0023.04ee.be64  15s    0x8001  0x4101   0x3D
Gi1/0/22  SA      32768     0023.04ee.be64   0s    0x8001  0x101    0x3D
Gi1/0/23  SA      32768     0023.04ee.be64  15s    0x8001  0x4103   0x3D
Gi1/0/24  SA      32768     0023.04ee.be64  20s    0x8001  0x103    0x3D
Gi2/0/21  SA      32768     0023.04ee.be64   0s    0x8001  0x102    0x3D
Gi2/0/22  SA      32768     0023.04ee.be64  15s    0x8001  0x4102   0x3D
Gi2/0/23  SA      32768     0023.04ee.be64  20s    0x8001  0x104    0x3D
Gi2/0/24  SA      32768     0023.04ee.be64  15s    0x8001  0x4104   0x3D

And after brining the peer keepalive back online:

show vpc
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 100
Peer status                     : peer adjacency formed ok
vPC keep-alive status           : peer is alive

HTH,

-ryan