[c-nsp] VRRP / MAC Forwarding Problem on Sup2/PFC2

[Sascha E. Pollok]

Hello people,

recently I have discussed a problem here and there and there
is not proper solution/explanation yet so I thought I'd share
it with you:

                      Server
                        |
                        |
                +-----3548XL-----+
     .1q Trunk  |                | .1q Trunk
                |      MST       |
                |                |
    MST Root  6509#1----------6509#2  MST Backup
  VRRP Backup   |    L2 Trunk    |    VRRP Master
    on SVI      |       .1q      |      on SVI
                |                |
              BB#1              BB#2   Backbone Routers


Problem: 6509#1 was VRRP Master before and terminated all the
IP traffic from the server on its SVI and routed it towards the
internet (via BB#1).

Now 6509#2 is configured VRRP master. Thus, because of MST,
the ethernet frames should travel on layer 2 to 6509#2 and
fall out of the SVI there like this:

   Server -> 3548XL -> 6509#1 -> 6509#2 -> SVI -> BB#2.

What actually happens is the strange part: the traffic
can still be seen on the SVI on 6509#1. 6509#1 is still
pushing the traffic from layer 2 to layer 3 on its local SVI
and the 6509#2 never sees the traffic coming from the server.

Even if I deconfigure VRRP on 6509#1. And: even if I do
"no ip address" on the SVI on 6509#1 it is still routing
the traffic (of course only incoming from the server as
the connected routes are gone when doing "no ip address").

The 6509#2 only sees the traffic in two cases:

   1. I shutdown the SVI on 6509#1. In this case the traffic
      is immediately switched to 6509#2 and routed to the 6509#2's
      SVI. When I "no shut" the SVI on 6509#1 we are back to the
      previous situation. Deleting and re-creating the SVI on
      6509#1 does not help.

   2. When I change the VRRP Group on 6509#2 to a VRRP Group that
      NEVER was configured on the SVI on 6509#1. Then the traffic
      goes as expected via the SVI on 6509#2.
      When I make 6509#1 the VRRP master in this situation, the
      L3 Traffic switches to 6509#1. When I make the 6509#2
      the master again, it does NOT switch back to 6509#2.

This leads me to the following assumption: the 6509#1 somehow
memorizes the destination MAC addresses that it is supposed to
"route" to the locally configured SVI even when the ethernet
frame's destination address is reachable via a remote trunk.

"sh mac-address-table vlan xx" on 6509#1 shows that it has been
dynamically learned via the trunk towards 6509#2. This looks
perfect. So there must be something else to let the 6509#1
send the packet to the SVI.

To make it more complex, I would like to add that a traceroute
from the server to the Internet looks like this:

    1  6509#2    (yes, thats correct)
    2  BB#1      (oh that can't be since there is
                  no link to BB#1 on 6509#2)
Details about the platform: 6509 with SUP2, MSFC2
and SFM cards running 12.2(18)SXF15.

Anyone?

Thank you for your thoughts.
Sascha