[c-nsp] IPv6 multicast on ethernet, weird problems

[Bernhard Schmidt]

Heya,

I'm trying to run sort of production (means native inside the backbone)
IPv6 multicast in a little network I'm working with and have some
seriously weird problems with IPv6 multicast on ethernet wires.

The setup looks similar to this:

         m6bone-Upstream (tunneled)
             |
             |
         Backbone 1 ----------- Backbone2 
	     |    `-------------, | |
	     |                  | | |
	     | ,----------------+-´ |
	     | |                |   |
	 Switch 1               Switch 2
	     |                      |
	     |                      |
	     `------- Access -------´

The two backbones are 7206 VXR NPE-G1 running 12.2(25)S2, the Access box
is a 7206 VXR NPE400 running various types of IOS. Switch1 is a 3500XL
12.0(5)WC9, and Switch2 a 2950G-48 with 12.1(19)EA1a. The setup is
running OSPFv3 for IGP and fulltable ipv6.multicast BGP on all boxes.

First of all, the access box was running 12.2(25)S2 as well. Setup
worked quite well except frequent annoying reloads when using any IPv6
multicast features. Due to the wish for IPv6-IPv6-tunnels I tried
12.3(11)T3 and had no luck with IPv6 multicast with symptoms I'll
describe in the next paragraph, so I went for 12.3(8)T6 and was happy
after all. The whole setup worked for more than a week without any
problems, there are a couple of downstreams on the access router testing
IPv6 multicast software. 

The problem I observed with 12.3(11)T3 had the following symptoms. A
downstream neighbor on the access router joined a multicast group with
many sources and quite some traffic, for example FF3E:20:2001:660::BEAC
(the m6bone testing beacon, ~10 sources and ~100kbps traffic). The
access router joined (*,G) upwards, thus Backbone 1 did this too and
very soon received the 100kbps on the RPT. The appropriate outbound
interface in "sh ipv6 mroute" is set. On the switch one could see the
(anticipated) port outbound rate of 100kbps plus some bps of unicast
traffic. But, on the incoming ethernet of the router there were only a
few kbps, the downstream did not receive traffic. Some times they were
lucky and one source (of ten) got through, creating the proper SPT
state and forwarding traffic, but still missing the rest of the sources.

Those very same problems suddenly appeared two days ago on two access
routers, the one I was talking about above (12.3(8)T6) and another one
with 12.2(25)S2 while a third one with 12.2(25)S2 worked without a flaw.
All of them are connected exactly the same way to both switches and are
in the same VLAN.

I tried reloading the box, I tried switching paths from Switch2 to
Switch1. RPF is definitely okay. The only thing that helped immediately
was a tunnel from the backbone to the access box instead of native
connection.

I will go to the POP in a few days and attach a sniffer, but any ideas
right now what could be wrong?

Bernhard