[c-nsp] Strange vlan behavior

[nachocheeze at gmail.com]

We've got a network I'm looking at that is predominately L2 switched;
a tangent of the old router-on-a-stick; some routing, but mostly
switching.  I fired up Wireshark on my laptop recently to diagnose
something, and noticed something a bit odd.  Here's a smaller version
of the network with a problem I can't quite figure out what is going
on.

HostX, HostY, and MyLaptop are all on the same L2 vlan / L3 IP network
(we'll say VLAN 31 and network 172.16.31.0/24).  Switches A,B,C, and D
are all lower end Cisco L2 only switches, Routers 1 and 2 are L2/L3
Catalyst 6500/MSFC.  Currently, the L3 SVI for VLAN 31 lives on Router
1, but I've tried moving it to Router 2 and the same problem keeps
happening.

All links are 802.1q trunks.  There's certain networks defined on
Router 1, and different networks that are defined on Router 2.
However, for some of those networks, there's hosts attached at
user-level switches at both "north" and "south" ends (yes, all the L2
vlans do span from end to end across every dotq trunk, and I *KNOW*
it's a bad design.  It was born of a specific necessity and needs to
change ASAP, but right now it isn't possible).  Router1 and Router2 in
addition to being fully trunked also have a dedicated numbered "routed
vlan" that is used to route the disparate user networks between them.

This is a scaled down version of the topology.

     HostX     HostY
     |              |
-----------------------------
       Switch D
        |
        Switch C
        |
        Router 1 (multiple vlans/SVIs)
        |
        Router 2 (multiple vlans/SVIs)
        |
        Switch B
        |
        Switch A
        |
        MyLaptop

What I noticed that is making no sense is the following; when sniffing
my network interface on MyLaptop, I can from time to time see snippets
of traffic that transit directly between HostX and HostY.  This is not
ARP (broadcast) traffic, or multicast traffic but direct station to
station unicast traffic between X and Y.  Not *all* their traffic,
like a SPAN port, but just little snippets here and there (sometimes a
few ICMP packets, sometimes a couple of HTTP packets, etc).  A sniff
of MyLaptop's NIC shows the source IP address / source MAC address of
HostX attempting a unicast transaction to the destination IP address /
destination MAC address of HostY.  Again, I'm seeing that unicast
transaction directly from my laptop's tcpdump from several trunk links
away.

I've checked this with other L2 end-user switches that are on the same
vlan/subnet in the north/south ends, and they all see this same kind
of issue too.  That means it's happening pretty much everywhere the
vlan is trunked, and possibly on other vlans.  From the way I
understand, apart from maybe some ARP traffic if HostA and HostB don't
know each other's L2 address, I should never see it; the traffic
between HostA and HostB should stay on Switch Y for their entire
conversation.

I've checked everything in the path between stations, and nothing that
I can find has been miscabled, no port monitoring is turned on
anywhere, etc.  Ideas for what I should start looking at? (besides a
total retrofit of the design; that's in the works.)