[c-nsp] TCAM troubles on 3750 stack

[Saku Ytti]

On (2006-04-29 12:30 +0200), Alexander Gall wrote:

> >> Path(0)AdjIP:130.59.36.9 Vl:1006 000a.f330.1d80 RWI:0x2
> 
> > Anything strange for 130.59.36.9? Did you have ARP for it? Was the MAC
> 
> Nothing strange about this next hop.  There are lots of other FIB
> entries with this adjancency that are perfectly fine, e.g.

Otoh RWI is different, which I believe is the pointer for final rewrite
information. So perhaps it's RWI 0x2 that is/was broken, or are the some
with RWI 0x2 that are/were working?

> Fib 160.53.0.0/16 Tbl:0 Bucket:52
>         Path(0)AdjIP:130.59.36.9 Vl:1007 000a.f330.1d80 RWI:0x3

> swiCP2#sh platform tcam utilization

I'd ask TAC to confirm that TCAM usage is ok.

> Maybe.  In fact, this router only has two uplinks and all FIB entries
> pointing to the same adjacency as the default route could be optimized
> away.  But I don't think this is what's happening.  The prefixes left
> in this list after "clear ip route *" seem to be arbitrary.  And
> anyway, why would they be listed as "failed routes", which appears to
> mean that they couldn't be programmed into the TCAM?  And why is the
> default route marked with "COVERING FIB ADJ Failed" in the FIB TCAM?

Covering FIB ADJ Failed, to me sounds like adjacency to the next-hop
is not ok, but I may be mistaken.

Thats quite odd, unless of course it has already timed out.

> Ah, yes, the address is there.  I admit that all of these sh platform
> commands have me very much confused (How can something called
> "mac-address" be L3 information?).  

Yeah it's bit odd, but somehow it's only L2 rewrite data for L3 lookups.

> > I guess there isn't routed port for it in SW2? Do you see it in 'show
> > platform mac-address-table'?

> > Can you see these in 'sh arp'?
> 
> No.  There are only "real" mac addresses in the arp table.  So far, I
> could only find the 00603 addresses in this tcam table mac-address
> table.  Here's the output of "sh platform forward" that I think is
> relevant for forwarding a packet from the router to the host, maybe
> you can make sense of it:
> 
> swiCP2#sh platform forward gigabitEthernet 2/0/4 0012.d9ba.40ca 0003.ba9b.07bb
> Ingress:
> Global Port Number: 56, lpn: 1 Asic Number: 2
> SASQUATCH ASIC type for asic num(2)
> Source Vlan Id: Real 138, Mapped 9. L2EncapType 0, L3EncapType 3
> Hashes: L2Src 0x05 L2Dst 0x0B L3Src 0x05 L3Dst 0x0B
>  Lookup                   Key-Used                  Index-Hit A-Data
> Classify 68_0E300003_BA9B07BB-00_00000012_D9BA40CA     00FFC 00000000
> InputACL 20_0E300003_BA9B07BB-00_00000012_D9BA40CA     01FF8 01000000
> L2LrnMsk FF_03FFFFFF_FFFFFFFF-00_000003FF_00000000
> L2Learn  83_00090012_D9BA40CA-C3_00002438_00000000     00850 00000060
> L2FwdMsk FF_03FFFFFF_FFFFFFFF
> L2Fwd    83_00090003_BA9B07BB                          00862 020100B3
> Station Descriptor: F0380007, DestIndex: F038, RewriteIndex: 0007
> SASQUATCH ASIC type for asic num(2)
> ==========================================
> Egress: Asic 2, switch 2
> Source Vlan Id: Real 138, Mapped 9. L2EncapType 0, L3EncapType 3
> portMap 0x2, non-SPAN portMap 0x2
> 
> Output Packets:
> ------------------------------------------
> GigabitEthernet2/0/4 Packet 1
>  Lookup                   Key-Used                  Index-Hit A-Data
> OutptACL 30_0E300003_BA9B07BB-06_00000012_D9BA40CA     01FFC 01000000
> Dropped due to failed deja vu check.

Aren't there any other output packets? Since this only displays output
interface GigabitEthernet2/0/4 which will not be used, as packet is coming
in from that interface.

> The RewriteIndex looks suspicious.  Slot 0007 is exactly the one in
> the "tcam table mac-address" that corresponds to 0006.03C8.A5F8.
> Coincidence? 

The 'mapped vlan' that can also be observed isn't any VLAN that I really
have. In 'simple TCAM switches' doing L3, lookup tables are heavily
abused to get plethora of functions out of them. This probably is
indication of one of those abuse, possibly some abuse related to using
IPv6? I really don't have enough clues to say what's going on :(. So If
you're opening TAC case and asking about these, please let the list know 
too. 
Do you happen to have more/complete packets what you captured? If possible
please send them as pcap to me, I'd like to take a peak.

I checked my production boxes and only found 0006.03 in boxes with IPv6,
after this I had to experiement to confirm this in lab:

LAB-SW1(config)#ipv6 unicast-routing 
LAB-SW1(config)#^Z
LAB-SW1#show platform tcam table mac-address detail | i 0006\.03
LAB-SW1#conf term
Enter configuration commands, one per line.  End with CNTL/Z.
LAB-SW1(config)#int vlan2
LAB-SW1(config-if)#ipv6 address 2001:6e9::1/64
LAB-SW1(config-if)#^Z
LAB-SW1#show platform tcam table mac-address detail | i 0006\.03
LAB-SW1#ping 2001:6e9::1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2001:6E9::1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 0/0/0 ms
LAB-SW1#show platform tcam table mac-address detail | i 0006\.03
  macAddress:                   0006.03BD.CC18
LAB-SW1#

I think this supports my suspicions, cisco is uing 0006.03 as some
sort of hack in rewrite information to signal IPv6 rewrite. It
only appears once you've passed IPv6 traffic.
Most probably this is not relevant to problem you're experiencing, but
quite interesting nevertheless.

> BTW, I can find 00-06-03 addresses in this particular TCAM table on
> our other 3750s as well.

-- 
  ++ytti