[c-nsp] TCAM troubles on 3750 stack
Alexander Gall
gall at switch.ch
Sat Apr 29 06:30:15 EDT 2006
On Sat, 29 Apr 2006 09:24:13 +0300, Saku Ytti <saku+cisco-nsp at ytti.fi> said:
> On (2006-04-28 21:49 +0200), Alexander Gall wrote:
>> swiCP2#sh platform ip unicast route
>> Dumping IOS-HL3U Fib info
>> Fib 0.0.0.0/0 Tbl:0 Bucket:0
>> Path(0)AdjIP:130.59.36.9 Vl:1006 000a.f330.1d80 RWI:0x2
>> HL3UFlags:0x28 COVERING FIB ADJ Failed
>> SFT Entry:hdl:0x3C HwFL:0x4
>> [...]
> Anything strange for 130.59.36.9? Did you have ARP for it? Was the MAC
Nothing strange about this next hop. There are lots of other FIB
entries with this adjancency that are perfectly fine, e.g.
Fib 160.53.0.0/16 Tbl:0 Bucket:52
Path(0)AdjIP:130.59.36.9 Vl:1007 000a.f330.1d80 RWI:0x3
HL3UFlags:0x80
SFT Entry:hdl:0x1C2 HwFL:0x4
> programmed correctly? This appears to be L3 interface, not SVI, right?
Yes, it's L3.
>> I don't know what exactly that means, but the effect was that all
>> traffic to destinations reached by the default route (this router
>> doesn't do BGP and uses a OSPF default route) was forwarded in
>> software. We're using the "desktop IPv4 and IPv6 default" SDM
>> template and none of the TCAMs is full. I finally figured out that
> How did you confirm that TCAM was not full?
swiCP2#sh platform tcam utilization
CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values
Unicast mac addresses: 672/5376 38/205
IPv4 IGMP groups: 144/1152 12/39
IPv4 unicast directly-connected routes: 672/5376 38/205
IPv4 unicast indirectly-connected routes: 144/1152 76/513
IPv6 Multicast groups: 672/5376 38/205
IPv6 unicast directly-connected routes: 672/5376 38/205
IPv6 unicast indirectly-connected routes: 128/1024 44/270
IPv4 policy based routing aces: 0/0 0/0
IPv4 qos aces: 512/512 6/6
IPv4 security aces: 1024/1024 868/868
IPv6 policy based routing aces: 0/0 0/0
IPv6 qos aces: 0/0 0/0
IPv6 security aces: 204/510 6/6
(this looks the same for all port ASICs)
>> However, the other stack member has the same problem. There would be
>> some other arbitrary prefix (sometimes several of them) that got stuck
>> in this state, e.g.
>>
>> swiCP2#remote command 2 sh platform ip unicast failed route
>> Total of 0 covering fib entries
>> Entries covered by Actual default route(0.0.0.0/0)
>> 195.176.224.0/19 Tbl:0 : Cover:0.0.0.0/0 Tbl:0
>> Total of 1 entries covered by 0.0.0.0/0 Tbl:0
> Is this really different route, or just optimization for TCAM? Could
> this be just 'link' in hardware, to really use default route for the
> more spesific?
Maybe. In fact, this router only has two uplinks and all FIB entries
pointing to the same adjacency as the default route could be optimized
away. But I don't think this is what's happening. The prefixes left
in this list after "clear ip route *" seem to be arbitrary. And
anyway, why would they be listed as "failed routes", which appears to
mean that they couldn't be programmed into the TCAM? And why is the
default route marked with "COVERING FIB ADJ Failed" in the FIB TCAM?
>> switch. No amount of mucking around with (d)CEF helped. Anybody seen
>> this or has any sort of clue what's going on?
>> For example, the MAC address of one of the affected hosts is
>> 0003.ba9b.07bb. The ethernet header of a packet captured with tcpdump
>> shows
>>
>> ETHER: ----- Ether Header -----
>> ETHER:
>> ETHER: Packet 78 arrived at 18:33:6.55
>> ETHER: Packet size = 98 bytes
>> ETHER: Destination = 0:6:3:c8:a5:f8,
>> ETHER: Source = 0:12:d9:ba:40:ca,
>> ETHER: Ethertype = 0800 (IP)
>> ETHER:
> Odd indeed, any chance that you've ad mistake in capturing or displaying
> the capture? 0:60:3e would be cisco. Do you have any 0060.3ec8.a5f8 in
> the network?
No. I'm pretty sure the capture is ok. It can't be coincidence that
0006.03c8.a5f8 shows up in the TCAM exactly like this either.
>> Where the heck does 0:6:3:c8:a5:f8 come from? This address doesn't
>> exist anywhere in our network. The arp cache and mac address table on
>> the switch (master and slave) are OK
> Did you try 'show mac-address-table' or 'show platform mac-addres-table'
> to locate the 0006.03c8.a5f8?
Yes, it's not there.
>> swiCP2#sh platform tcam table mac-address | inc BB
>> 7 B0090003 BA9B07BB
> I believe this command only displays L3 information. Use 'show platform
> mac-address-table' not tcam table.
Ah, yes, the address is there. I admit that all of these sh platform
commands have me very much confused (How can something called
"mac-address" be L3 information?).
>> But there's no trace of 0003.ba9b.07bb in the mac TCAM of switch #2
>>
>> swiCP2#remote command 2 sh platform tcam table mac-address | inc BB
>> Switch : 2 :
>> ------------
> I guess there isn't routed port for it in SW2? Do you see it in 'show
> platform mac-address-table'?
>> swiCP2#remote command 2 sh platform tcam table mac-address | inc 0006
>> Switch : 2 :
>> ------------
>> 2 F00C0006 03C8C818
>> 3 C00D0006 03C8C678
>> 4 90090006 03C8AFB8
>> 6 90090006 03C8B638
>> 7 A0090006 03C8A5F8
>> 9 F0090006 03C8C338
>> 10 A0090006 03C89C38
>> 11 A0090006 03C8B978
>> 13 A0040006 03C8A2B8
>> 14 90040006 03C89758
>> 17 A0040006 03C8BCB8
>> 18 90090006 03C89F78
> Can you see these in 'sh arp'?
No. There are only "real" mac addresses in the arp table. So far, I
could only find the 00603 addresses in this tcam table mac-address
table. Here's the output of "sh platform forward" that I think is
relevant for forwarding a packet from the router to the host, maybe
you can make sense of it:
swiCP2#sh platform forward gigabitEthernet 2/0/4 0012.d9ba.40ca 0003.ba9b.07bb
Ingress:
Global Port Number: 56, lpn: 1 Asic Number: 2
SASQUATCH ASIC type for asic num(2)
Source Vlan Id: Real 138, Mapped 9. L2EncapType 0, L3EncapType 3
Hashes: L2Src 0x05 L2Dst 0x0B L3Src 0x05 L3Dst 0x0B
Lookup Key-Used Index-Hit A-Data
Classify 68_0E300003_BA9B07BB-00_00000012_D9BA40CA 00FFC 00000000
InputACL 20_0E300003_BA9B07BB-00_00000012_D9BA40CA 01FF8 01000000
L2LrnMsk FF_03FFFFFF_FFFFFFFF-00_000003FF_00000000
L2Learn 83_00090012_D9BA40CA-C3_00002438_00000000 00850 00000060
L2FwdMsk FF_03FFFFFF_FFFFFFFF
L2Fwd 83_00090003_BA9B07BB 00862 020100B3
Station Descriptor: F0380007, DestIndex: F038, RewriteIndex: 0007
SASQUATCH ASIC type for asic num(2)
==========================================
Egress: Asic 2, switch 2
Source Vlan Id: Real 138, Mapped 9. L2EncapType 0, L3EncapType 3
portMap 0x2, non-SPAN portMap 0x2
Output Packets:
------------------------------------------
GigabitEthernet2/0/4 Packet 1
Lookup Key-Used Index-Hit A-Data
OutptACL 30_0E300003_BA9B07BB-06_00000012_D9BA40CA 01FFC 01000000
Dropped due to failed deja vu check.
The RewriteIndex looks suspicious. Slot 0007 is exactly the one in
the "tcam table mac-address" that corresponds to 0006.03C8.A5F8.
Coincidence?
BTW, I can find 00-06-03 addresses in this particular TCAM table on
our other 3750s as well.
--
Alex
More information about the cisco-nsp
mailing list