[c-nsp] TCAM troubles on 3750 stack

[Alexander Gall]

It got worse.  I had to perform an emergency reload after the router
had started dropping all traffic routed via the default route (no time
to investigate this new effect, though).  The symptom with the wrong
mac address is now gone, but the "covering fib" effect is still there.
This box seems truly broken.  Comments inline.

On Sat, 29 Apr 2006 14:52:24 +0300, Saku Ytti <saku+cisco-nsp at ytti.fi> said:

> On (2006-04-29 12:30 +0200), Alexander Gall wrote:
>> >> Path(0)AdjIP:130.59.36.9 Vl:1006 000a.f330.1d80 RWI:0x2
>> 
>> > Anything strange for 130.59.36.9? Did you have ARP for it? Was the MAC
>> 
>> Nothing strange about this next hop.  There are lots of other FIB
>> entries with this adjancency that are perfectly fine, e.g.

> Otoh RWI is different, which I believe is the pointer for final rewrite
> information. So perhaps it's RWI 0x2 that is/was broken, or are the some
> with RWI 0x2 that are/were working?

Yes, all more specific routes with the same adjancency have the same
RWI value and they work.  Clearly, the adjancency itself is fine.

>> Maybe.  In fact, this router only has two uplinks and all FIB entries
>> pointing to the same adjacency as the default route could be optimized
>> away.  But I don't think this is what's happening.  The prefixes left
>> in this list after "clear ip route *" seem to be arbitrary.  And
>> anyway, why would they be listed as "failed routes", which appears to
>> mean that they couldn't be programmed into the TCAM?  And why is the
>> default route marked with "COVERING FIB ADJ Failed" in the FIB TCAM?

> Covering FIB ADJ Failed, to me sounds like adjacency to the next-hop
> is not ok, but I may be mistaken.

But the same adjancency is just fine for all other routes pointing to
the same p2p link.  It doesn't make sense to me.

>> swiCP2#sh platform forward gigabitEthernet 2/0/4 0012.d9ba.40ca 0003.ba9b.07bb

[...]

> Aren't there any other output packets? Since this only displays output
> interface GigabitEthernet2/0/4 which will not be used, as packet is coming
> in from that interface.

Oops, I had picked the wrong interface :-(

> The 'mapped vlan' that can also be observed isn't any VLAN that I really
> have. In 'simple TCAM switches' doing L3, lookup tables are heavily
> abused to get plethora of functions out of them. This probably is
> indication of one of those abuse, possibly some abuse related to using
> IPv6? I really don't have enough clues to say what's going on :(. So If
> you're opening TAC case and asking about these, please let the list know 
> too. 
> Do you happen to have more/complete packets what you captured? If possible
> please send them as pcap to me, I'd like to take a peak.

Unfortunately, I didn't save the packet traces and the effect is gone
since the emergency reload.

> I checked my production boxes and only found 0006.03 in boxes with IPv6,
> after this I had to experiement to confirm this in lab:

> LAB-SW1(config)#ipv6 unicast-routing 
> LAB-SW1(config)#^Z
> LAB-SW1#show platform tcam table mac-address detail | i 0006\.03
> LAB-SW1#conf term
> Enter configuration commands, one per line.  End with CNTL/Z.
> LAB-SW1(config)#int vlan2
> LAB-SW1(config-if)#ipv6 address 2001:6e9::1/64
> LAB-SW1(config-if)#^Z
> LAB-SW1#show platform tcam table mac-address detail | i 0006\.03
> LAB-SW1#ping 2001:6e9::1

> Type escape sequence to abort.
> Sending 5, 100-byte ICMP Echos to 2001:6E9::1, timeout is 2 seconds:
> !!!!!
> Success rate is 100 percent (5/5), round-trip min/avg/max = 0/0/0 ms
> LAB-SW1#show platform tcam table mac-address detail | i 0006\.03
>   macAddress:                   0006.03BD.CC18
> LAB-SW1#

> I think this supports my suspicions, cisco is uing 0006.03 as some
> sort of hack in rewrite information to signal IPv6 rewrite. It
> only appears once you've passed IPv6 traffic.

Interesting.  We have IPv6 enabled in all subnets in question.

> Most probably this is not relevant to problem you're experiencing, but
> quite interesting nevertheless.

I'd take the fact that it's precisely these special mac addresses that
appear in forwarded packets as an indication that this is in fact
relevant, i.e. it could be a bug in the code that programs the
hardware for IPv6.

Thanks for your comments and suggestions, 

--
Alex