[j-nsp] TCAM full on EX8200?
Michele Bergonzoni
bergonz at labs.it
Wed Oct 12 12:40:55 EDT 2011
THE SHORT QUESTION:
How can I see how full my IPv4 FIB is, on an EX8200 with EX8200-40XS
linecards and 11.3R2.4 ? I can connect to fpc and give the show
commands, but I need help interpreting the results.
If it actually turns out to be full, is there something I should do to
support the 500K routes that I see in the data sheet? I mean e.g.
something like the sdm templates or "mls cef maximum-routes" in the
cisco boxes?
THE LONG, PAINFUL STORY:
I have a pair of such machines, configured to take IPv4 and IPv6 full
routing table in a routing instance.
The datasheets say that I should be able to take 500.000 IPv4 and
250.000 IPv6 routes, and now I have 370K and 7K active routes:
internet.inet.0: 370806 destinations, 741349 routes (370798 active, 0
holddown, 8 hidden)
internet.inet6.0: 7161 destinations, 7168 routes (7161 active, 0
holddown, 0 hidden)
I am experiencing issues where traffic to some destinations is lost
while other destinations work, so I would like to see if the TCAM
utilization is OK. The issue was present also with some 10.x releases I
tried before, so it is not specific to this release.
I tried to open a session to the fpc with "vty fpc2", and it prints log
lines like this:
[Wed Oct 12 15:39:48 2011 LOG: Err] Failed to Add IPv4 Uc prefix/len
[PUBLIC IPv4 PREFIX HERE] (cstatus: 65565)
[Wed Oct 12 15:39:48 2011 LOG: Err] Failed to h/w update ip uc route
entry (status: 22)
[Wed Oct 12 15:39:48 2011 LOG: Err] Failed to install the RT entry
(status: 22)
[Wed Oct 12 15:39:48 2011 LOG: Err] RT-HAL,rt_entry_add_msg_proc,2378:
rt_halp_vectors->rt_create failed
[Wed Oct 12 15:39:48 2011 LOG: Err] RT-HAL,rt_entry_add_msg_proc,2428:
proto ipv4,len 24 prefix [THE SAME PUBLIC IPv4 PREFIX] nh 131070
[Wed Oct 12 15:39:48 2011 LOG: Err] RT-HAL,rt_msg_handler,580: route
process failed
And so on for more IPv4 prefixes. Similar lines are also in messages.
This looks very much like a TCAM full error to me, but TCAM utilization
doesn't seem anywhere full. This is the (cleaned) transcript of my
attempt to see the TCAM utilization:
PFEM2(vty)# show tcam vendor
Vendor = netlogic_frs Vendor_id = 6
Vendor = Lion Internal TCAM Vendor_id = 7
PFEM2(vty)# show tcam vendor 6 instances
Vendor Instance Page Size
--------------------------------------------
netlogic_frs 0 4
netlogic_frs 1 4
netlogic_frs 2 4
netlogic_frs 3 4
netlogic_frs 4 4
PFEM2(vty)# show tcam vendor 7 instances
Vendor Instance Page Size
--------------------------------------------
Lion Internal TCAM 4 4
Lion Internal TCAM 5 4
Lion Internal TCAM 6 4
PFEM2(vty)# show tcam vendor 6 instance 0 rules
Instance 0
DB 0 Ingr L2COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 1 Ingr PACL: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 2 Ingr VACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 3 Ingr L3COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 4 Ingr RACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 5 Egr PACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 6 Egr PCL1: 0/ 508 (current/alloc) rules. Util.
0.000%
DB 7 Ingr L2COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 8 Ingr PACL-v6: 0/ 256 (current/alloc) rules. Util.
0.000%
DB 9 Ingr VACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 10 Egr PACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 11 Egr PCL1-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 12 Ingr L3COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 13 Ingr L3DYN-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 14 Ingr RACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 15 Ingr L2DYN: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 16 Ingr L2COS MD: 5/ 28 (current/alloc) rules. Util.
17.857%
DB 17 Ingr L2COS RVI: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 18 Ingr L2COS RVIALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 19 Ingr L2COS L2: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 20 Ingr L2COS L2ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 21 Ingr L2COS DEF: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 22 Ingr L3COS MD: 4/ 32 (current/alloc) rules. Util.
12.500%
DB 23 Ingr L3COS L3: 0/ 960 (current/alloc) rules. Util.
0.000%
DB 24 Ingr L3COS L3ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 25 Ingr L3COS DEF: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 26 Ingr L2COS MD-v6: 5/ 20 (current/alloc) rules. Util.
25.000%
DB 27 Ingr L2COS RVI-v6: 0/ 208 (current/alloc) rules. Util.
0.000%
DB 28 Ingr L2COS RA-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 29 Ingr L2COS L2-v6: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 30 Ingr L2COS L2A-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 31 Ingr L2COS DEF-v6: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 32 Ingr L3COS MD-v6: 4/ 20 (current/alloc) rules. Util.
20.000%
DB 33 Ingr L3COS L3-v6: 0/ 468 (current/alloc) rules. Util.
0.000%
DB 34 Ingr L3COS L3ALL-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 35 Ingr L3COS DEF-v6: 2/ 4 (current/alloc) rules. Util.
50.000%
Pool1 members expandable by 16 blocks
Pool2 members expandable by 29 blocks
Block = 1k 40 Byte rules = 512 80 byte rules
PFEM2(vty)# show tcam vendor 6 instance 1 rules
Instance 1
DB 0 Ingr L2COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 1 Ingr PACL: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 2 Ingr VACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 3 Ingr L3COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 4 Ingr RACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 5 Egr PACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 6 Egr PCL1: 0/ 508 (current/alloc) rules. Util.
0.000%
DB 7 Ingr L2COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 8 Ingr PACL-v6: 0/ 256 (current/alloc) rules. Util.
0.000%
DB 9 Ingr VACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 10 Egr PACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 11 Egr PCL1-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 12 Ingr L3COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 13 Ingr L3DYN-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 14 Ingr RACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 15 Ingr L2DYN: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 16 Ingr L2COS MD: 5/ 28 (current/alloc) rules. Util.
17.857%
DB 17 Ingr L2COS RVI: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 18 Ingr L2COS RVIALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 19 Ingr L2COS L2: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 20 Ingr L2COS L2ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 21 Ingr L2COS DEF: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 22 Ingr L3COS MD: 4/ 32 (current/alloc) rules. Util.
12.500%
DB 23 Ingr L3COS L3: 0/ 960 (current/alloc) rules. Util.
0.000%
DB 24 Ingr L3COS L3ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 25 Ingr L3COS DEF: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 26 Ingr L2COS MD-v6: 5/ 20 (current/alloc) rules. Util.
25.000%
DB 27 Ingr L2COS RVI-v6: 0/ 208 (current/alloc) rules. Util.
0.000%
DB 28 Ingr L2COS RA-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 29 Ingr L2COS L2-v6: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 30 Ingr L2COS L2A-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 31 Ingr L2COS DEF-v6: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 32 Ingr L3COS MD-v6: 4/ 20 (current/alloc) rules. Util.
20.000%
DB 33 Ingr L3COS L3-v6: 0/ 468 (current/alloc) rules. Util.
0.000%
DB 34 Ingr L3COS L3ALL-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 35 Ingr L3COS DEF-v6: 2/ 4 (current/alloc) rules. Util.
50.000%
PFEM2(vty)# show tcam vendor 6 instance 2 rules
Instance 2
DB 0 Ingr L2COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 1 Ingr PACL: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 2 Ingr VACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 3 Ingr L3COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 4 Ingr RACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 5 Egr PACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 6 Egr PCL1: 0/ 508 (current/alloc) rules. Util.
0.000%
DB 7 Ingr L2COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 8 Ingr PACL-v6: 0/ 256 (current/alloc) rules. Util.
0.000%
DB 9 Ingr VACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 10 Egr PACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 11 Egr PCL1-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 12 Ingr L3COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 13 Ingr L3DYN-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 14 Ingr RACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 15 Ingr L2DYN: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 16 Ingr L2COS MD: 5/ 28 (current/alloc) rules. Util.
17.857%
DB 17 Ingr L2COS RVI: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 18 Ingr L2COS RVIALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 19 Ingr L2COS L2: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 20 Ingr L2COS L2ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 21 Ingr L2COS DEF: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 22 Ingr L3COS MD: 4/ 32 (current/alloc) rules. Util.
12.500%
DB 23 Ingr L3COS L3: 0/ 960 (current/alloc) rules. Util.
0.000%
DB 24 Ingr L3COS L3ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 25 Ingr L3COS DEF: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 26 Ingr L2COS MD-v6: 5/ 20 (current/alloc) rules. Util.
25.000%
DB 27 Ingr L2COS RVI-v6: 0/ 208 (current/alloc) rules. Util.
0.000%
DB 28 Ingr L2COS RA-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 29 Ingr L2COS L2-v6: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 30 Ingr L2COS L2A-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 31 Ingr L2COS DEF-v6: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 32 Ingr L3COS MD-v6: 4/ 20 (current/alloc) rules. Util.
20.000%
DB 33 Ingr L3COS L3-v6: 0/ 468 (current/alloc) rules. Util.
0.000%
DB 34 Ingr L3COS L3ALL-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 35 Ingr L3COS DEF-v6: 2/ 4 (current/alloc) rules. Util.
50.000%
PFEM2(vty)# show tcam vendor 6 instance 3 rules
Instance 3
DB 0 Ingr L2COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 1 Ingr PACL: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 2 Ingr VACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 3 Ingr L3COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 4 Ingr RACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 5 Egr PACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 6 Egr PCL1: 0/ 508 (current/alloc) rules. Util.
0.000%
DB 7 Ingr L2COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 8 Ingr PACL-v6: 0/ 256 (current/alloc) rules. Util.
0.000%
DB 9 Ingr VACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 10 Egr PACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 11 Egr PCL1-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 12 Ingr L3COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 13 Ingr L3DYN-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 14 Ingr RACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 15 Ingr L2DYN: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 16 Ingr L2COS MD: 5/ 28 (current/alloc) rules. Util.
17.857%
DB 17 Ingr L2COS RVI: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 18 Ingr L2COS RVIALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 19 Ingr L2COS L2: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 20 Ingr L2COS L2ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 21 Ingr L2COS DEF: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 22 Ingr L3COS MD: 4/ 32 (current/alloc) rules. Util.
12.500%
DB 23 Ingr L3COS L3: 0/ 960 (current/alloc) rules. Util.
0.000%
DB 24 Ingr L3COS L3ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 25 Ingr L3COS DEF: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 26 Ingr L2COS MD-v6: 5/ 20 (current/alloc) rules. Util.
25.000%
DB 27 Ingr L2COS RVI-v6: 0/ 208 (current/alloc) rules. Util.
0.000%
DB 28 Ingr L2COS RA-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 29 Ingr L2COS L2-v6: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 30 Ingr L2COS L2A-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 31 Ingr L2COS DEF-v6: 2/ 4 (current/alloc) rules. Util.
50.000%
DB 32 Ingr L3COS MD-v6: 4/ 20 (current/alloc) rules. Util.
20.000%
DB 33 Ingr L3COS L3-v6: 0/ 468 (current/alloc) rules. Util.
0.000%
DB 34 Ingr L3COS L3ALL-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 35 Ingr L3COS DEF-v6: 2/ 4 (current/alloc) rules. Util.
50.000%
PFEM2(vty)# show tcam vendor 6 instance 4 rules
Instance 4
DB 0 Ingr L2COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 1 Ingr PACL: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 2 Ingr VACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 3 Ingr L3COS: 0/ 1024 (current/alloc) rules. Util.
0.000%
DB 4 Ingr RACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 5 Egr PACL: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 6 Egr PCL1: 8/ 508 (current/alloc) rules. Util.
1.575%
DB 7 Ingr L2COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 8 Ingr PACL-v6: 0/ 256 (current/alloc) rules. Util.
0.000%
DB 9 Ingr VACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 10 Egr PACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 11 Egr PCL1-v6: 7/ 512 (current/alloc) rules. Util.
1.367%
DB 12 Ingr L3COS-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 13 Ingr L3DYN-v6: 0/ 512 (current/alloc) rules. Util.
0.000%
DB 14 Ingr RACL-v6: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 15 Ingr L2DYN: 0/ 0 (current/alloc) rules. Util.
-1.000%
DB 16 Ingr L2COS MD: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 17 Ingr L2COS RVI: 0/ 696 (current/alloc) rules. Util.
0.000%
DB 18 Ingr L2COS RVIALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 19 Ingr L2COS L2: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 20 Ingr L2COS L2ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 21 Ingr L2COS DEF: 0/ 4 (current/alloc) rules. Util.
0.000%
DB 22 Ingr L3COS MD: 0/ 32 (current/alloc) rules. Util.
0.000%
DB 23 Ingr L3COS L3: 0/ 960 (current/alloc) rules. Util.
0.000%
DB 24 Ingr L3COS L3ALL: 0/ 28 (current/alloc) rules. Util.
0.000%
DB 25 Ingr L3COS DEF: 0/ 4 (current/alloc) rules. Util.
0.000%
DB 26 Ingr L2COS MD-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 27 Ingr L2COS RVI-v6: 0/ 208 (current/alloc) rules. Util.
0.000%
DB 28 Ingr L2COS RA-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 29 Ingr L2COS L2-v6: 0/ 240 (current/alloc) rules. Util.
0.000%
DB 30 Ingr L2COS L2A-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 31 Ingr L2COS DEF-v6: 0/ 4 (current/alloc) rules. Util.
0.000%
DB 32 Ingr L3COS MD-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 33 Ingr L3COS L3-v6: 0/ 468 (current/alloc) rules. Util.
0.000%
DB 34 Ingr L3COS L3ALL-v6: 0/ 20 (current/alloc) rules. Util.
0.000%
DB 35 Ingr L3COS DEF-v6: 0/ 4 (current/alloc) rules. Util.
0.000%
The other instances of vendor 6 are similar. Instances of vendor 7 give
no information, showing only zeroes. The number shown are not giving me
any clue.
Besides, if it actually is a TCAM full issue, why is it happening? The
box with these cards should support 500K IPv4 routes, am I doing
something wrong? Should I repartition the TCAM in some way, as I'm used
to do with cisco boxes? I find no mention of this kind of configuration
in the docs.
Thanks in advance for your kind replies.
Bergonz
--
Ing. Michele Bergonzoni - Laboratori Guglielmo Marconi S.p.a.
Phone:+39-051-4392826 e-mail: bergonz at labs.it
alt.advanced.networks.design.configure.operate
More information about the juniper-nsp
mailing list