[c-nsp] Strange VSS Problem

[Andrew Cruse]

Hoping someone can point me in the right direction on this...

Hardware details are pasted at the end for the curious.  Noticed a pretty
consistent 25% packet-loss to *some* though not all destinations routed a
pair of VSS 6509's.  For example, no packet-loss pinging google.com, but 25%
loss pinging an interface on the router itself, 25% loss to Yahoo.  A little
tough to isolate, but ultimately we were able to determine that loss was
coming from the 6500's.

Digging further, it appears to be a MAC synchronization issue of some kind.

Check for a MAC address of an adjacent switch, I expect to see:

switch#sh mac-address-table address 0017.9431.05c0 all
  vlan   mac address     type    learn     age              ports
------+----------------+--------+-----+----------+--------------------------
Supervisor switch 1 Module 5
     1  0017.9431.05c0   dynamic  Yes        240   Gi2/7/45
switch 1 Module 7:
     1  0017.9431.05c0   dynamic  Yes        245   Gi2/7/45
Supervisor switch 2 Module 5
     1  0017.9431.05c0   dynamic  Yes        240   Gi2/7/45
switch 2 Module 7:
*    1  0017.9431.05c0   dynamic  Yes          0   Gi2/7/45

However issuing the command repeatedly, occasionally turns up these results:

switch#sh mac-address-table address 0017.9431.05c0 all
Legend: * - primary entry
        age - seconds since last seen
        n/a - not available

  vlan   mac address     type    learn     age              ports
------+----------------+--------+-----+----------+--------------------------
switch 1 Module 7:
     1  0017.9431.05c0   dynamic  Yes         15   Gi2/7/45
switch 2 Module 7:
*    1  0017.9431.05c0   dynamic  Yes          0   Gi2/7/45

Yikes!  The MAC address is now missing from the Sup cards.  That condition
seems to last only a split second, but certainly that would seem to be the
problem.  Cisco has this to say:


Configure the MAC aging timer to three times the MAC synchronization timer
value.

The default MAC synchronization and MAC aging timers can cause unknown
unicast flooding. VSS can cause traffic to flow asymmetrically such that the
source MAC address is only learned on one chassis. The MAC aging timer of
300 seconds and MAC synchronization timer of 160 seconds allows for up to 20
seconds of unknown unicast flooding for any given MAC address in a 320
second interval. In order to resolve this, change the timers such that the
aging timer is three times as long as synchronization timer, for
example, *mac-address-table
aging-time<http://www.cisco.com/en/US/docs/ios/12_3t/lanswitch/command/reference/lan_m1gt.html#wp1019701>
480* .
Makes sense, but we already have the aging time set to 3 times the
synchronization timer value:

switch#sh mac-address synchronize stat
<snip>
Global Status:
Status of feature enabled on the switch             :  on
Default activity time                               :  160
Configured current activity time                    :  160

switch#sh mac-address-table aging-time
Vlan    Aging Time
----    ----------
Global  480
no vlan age other than global age configured

I suppose I could take the aging time up to 640 but I have no reason to
believe 4x is any better than 3x.  Has anyone else seen this or have any
ideas?  Hardware/software details at bottom of email.

Thanks,

Andrew

6500 VSS as follows:
Version 12.2(33)SXI4

#sh mod
Mod Ports Card Type                              Model              Serial
No.
--- ----- -------------------------------------- ------------------
-----------
  5    5  Supervisor Engine 720 10GE (Active)    VS-S720-10G
  7   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX

Mod MAC addresses                       Hw    Fw           Sw
Status
--- ---------------------------------- ------ ------------ ------------
-------
  5  001e.4a7f.0010 to 001e.4a7f.0017   3.2   8.5(3)       12.2(33)SXI4 Ok
  7  0026.0bda.22c0 to 0026.0bda.22ef   3.2   12.2(18r)S1  12.2(33)SXI4 Ok

Mod  Sub-Module                  Model              Serial       Hw
Status
---- --------------------------- ------------------ ----------- -------
-------
  5  Policy Feature Card 3       VS-F6K-PFC3CXL     SAL1414EFEB  1.2    Ok
  5  MSFC3 Daughterboard         VS-F6K-MSFC3       SAL1417GLWF  2.1    Ok
  7  Distributed Forwarding Card WS-F6700-DFC3CXL   SAL11488MT2  1.0    Ok

switch#sh mac-address-table address 0017.9431.05c0 all
Legend: * - primary entry
        age - seconds since last seen
        n/a - not available