[j-nsp] TCAM full on EX8200?

Michele Bergonzoni bergonz at labs.it
Wed Oct 12 12:40:55 EDT 2011


THE SHORT QUESTION:

How can I see how full my IPv4 FIB is, on an EX8200 with EX8200-40XS 
linecards and 11.3R2.4 ? I can connect to fpc and give the show 
commands, but I need help interpreting the results.

If it actually turns out to be full, is there something I should do to 
support the 500K routes that I see in the data sheet? I mean e.g. 
something like the sdm templates or "mls cef maximum-routes" in the 
cisco boxes?

THE LONG, PAINFUL STORY:

I have a pair of such machines, configured to take IPv4 and IPv6 full 
routing table in a routing instance.

The datasheets say that I should be able to take 500.000 IPv4 and 
250.000 IPv6 routes, and now I have 370K and 7K active routes:

internet.inet.0: 370806 destinations, 741349 routes (370798 active, 0 
holddown, 8 hidden)

internet.inet6.0: 7161 destinations, 7168 routes (7161 active, 0 
holddown, 0 hidden)

I am experiencing issues where traffic to some destinations is lost 
while other destinations work, so I would like to see if the TCAM 
utilization is OK. The issue was present also with some 10.x releases I 
tried before, so it is not specific to this release.

I tried to open a session to the fpc with "vty fpc2", and it prints log 
lines like this:

[Wed Oct 12 15:39:48 2011 LOG: Err] Failed to Add IPv4 Uc prefix/len 
[PUBLIC IPv4 PREFIX HERE] (cstatus: 65565)
[Wed Oct 12 15:39:48 2011 LOG: Err] Failed to h/w update ip uc route 
entry (status: 22)
[Wed Oct 12 15:39:48 2011 LOG: Err] Failed to install the RT entry 
(status: 22)
[Wed Oct 12 15:39:48 2011 LOG: Err] RT-HAL,rt_entry_add_msg_proc,2378: 
rt_halp_vectors->rt_create failed
[Wed Oct 12 15:39:48 2011 LOG: Err] RT-HAL,rt_entry_add_msg_proc,2428: 
proto ipv4,len 24 prefix [THE SAME PUBLIC IPv4 PREFIX] nh 131070
[Wed Oct 12 15:39:48 2011 LOG: Err] RT-HAL,rt_msg_handler,580: route 
process failed

And so on for more IPv4 prefixes. Similar lines are also in messages. 
This looks very much like a TCAM full error to me, but TCAM utilization 
doesn't seem anywhere full. This is the (cleaned) transcript of my 
attempt to see the TCAM utilization:

PFEM2(vty)# show tcam vendor
Vendor = netlogic_frs Vendor_id = 6
Vendor = Lion Internal TCAM Vendor_id = 7

PFEM2(vty)# show tcam vendor 6 instances

  Vendor         Instance        Page Size
--------------------------------------------
  netlogic_frs         0         4
  netlogic_frs         1         4
  netlogic_frs         2         4
  netlogic_frs         3         4
  netlogic_frs         4         4
PFEM2(vty)# show tcam vendor 7 instances

  Vendor         Instance        Page Size
--------------------------------------------
  Lion Internal TCAM         4         4
  Lion Internal TCAM         5         4
  Lion Internal TCAM         6         4

PFEM2(vty)# show tcam vendor 6 instance 0 rules
Instance 0
   DB  0         Ingr L2COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  1          Ingr PACL:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB  2          Ingr VACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  3         Ingr L3COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  4          Ingr RACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  5           Egr PACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  6           Egr PCL1:     0/   508 (current/alloc) rules. Util. 
0.000%
   DB  7      Ingr L2COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB  8       Ingr PACL-v6:     0/   256 (current/alloc) rules. Util. 
0.000%
   DB  9       Ingr VACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 10        Egr PACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 11        Egr PCL1-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 12      Ingr L3COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 13      Ingr L3DYN-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 14       Ingr RACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 15         Ingr L2DYN:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 16      Ingr L2COS MD:     5/    28 (current/alloc) rules. Util. 
17.857%
   DB 17     Ingr L2COS RVI:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB 18  Ingr L2COS RVIALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 19      Ingr L2COS L2:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 20   Ingr L2COS L2ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 21     Ingr L2COS DEF:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 22      Ingr L3COS MD:     4/    32 (current/alloc) rules. Util. 
12.500%
   DB 23      Ingr L3COS L3:     0/   960 (current/alloc) rules. Util. 
0.000%
   DB 24   Ingr L3COS L3ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 25     Ingr L3COS DEF:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 26   Ingr L2COS MD-v6:     5/    20 (current/alloc) rules. Util. 
25.000%
   DB 27  Ingr L2COS RVI-v6:     0/   208 (current/alloc) rules. Util. 
0.000%
   DB 28   Ingr L2COS RA-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 29   Ingr L2COS L2-v6:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 30  Ingr L2COS L2A-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 31  Ingr L2COS DEF-v6:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 32   Ingr L3COS MD-v6:     4/    20 (current/alloc) rules. Util. 
20.000%
   DB 33   Ingr L3COS L3-v6:     0/   468 (current/alloc) rules. Util. 
0.000%
   DB 34 Ingr L3COS L3ALL-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 35  Ingr L3COS DEF-v6:     2/     4 (current/alloc) rules. Util. 
50.000%

           Pool1 members expandable by 16 blocks
           Pool2 members expandable by 29 blocks
           Block = 1k 40 Byte rules = 512 80 byte rules
PFEM2(vty)# show tcam vendor 6 instance 1 rules
Instance 1
   DB  0         Ingr L2COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  1          Ingr PACL:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB  2          Ingr VACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  3         Ingr L3COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  4          Ingr RACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  5           Egr PACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  6           Egr PCL1:     0/   508 (current/alloc) rules. Util. 
0.000%
   DB  7      Ingr L2COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB  8       Ingr PACL-v6:     0/   256 (current/alloc) rules. Util. 
0.000%
   DB  9       Ingr VACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 10        Egr PACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 11        Egr PCL1-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 12      Ingr L3COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 13      Ingr L3DYN-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 14       Ingr RACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 15         Ingr L2DYN:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 16      Ingr L2COS MD:     5/    28 (current/alloc) rules. Util. 
17.857%
   DB 17     Ingr L2COS RVI:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB 18  Ingr L2COS RVIALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 19      Ingr L2COS L2:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 20   Ingr L2COS L2ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 21     Ingr L2COS DEF:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 22      Ingr L3COS MD:     4/    32 (current/alloc) rules. Util. 
12.500%
   DB 23      Ingr L3COS L3:     0/   960 (current/alloc) rules. Util. 
0.000%
   DB 24   Ingr L3COS L3ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 25     Ingr L3COS DEF:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 26   Ingr L2COS MD-v6:     5/    20 (current/alloc) rules. Util. 
25.000%
   DB 27  Ingr L2COS RVI-v6:     0/   208 (current/alloc) rules. Util. 
0.000%
   DB 28   Ingr L2COS RA-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 29   Ingr L2COS L2-v6:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 30  Ingr L2COS L2A-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 31  Ingr L2COS DEF-v6:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 32   Ingr L3COS MD-v6:     4/    20 (current/alloc) rules. Util. 
20.000%
   DB 33   Ingr L3COS L3-v6:     0/   468 (current/alloc) rules. Util. 
0.000%
   DB 34 Ingr L3COS L3ALL-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 35  Ingr L3COS DEF-v6:     2/     4 (current/alloc) rules. Util. 
50.000%

PFEM2(vty)# show tcam vendor 6 instance 2 rules
Instance 2
   DB  0         Ingr L2COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  1          Ingr PACL:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB  2          Ingr VACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  3         Ingr L3COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  4          Ingr RACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  5           Egr PACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  6           Egr PCL1:     0/   508 (current/alloc) rules. Util. 
0.000%
   DB  7      Ingr L2COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB  8       Ingr PACL-v6:     0/   256 (current/alloc) rules. Util. 
0.000%
   DB  9       Ingr VACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 10        Egr PACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 11        Egr PCL1-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 12      Ingr L3COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 13      Ingr L3DYN-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 14       Ingr RACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 15         Ingr L2DYN:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 16      Ingr L2COS MD:     5/    28 (current/alloc) rules. Util. 
17.857%
   DB 17     Ingr L2COS RVI:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB 18  Ingr L2COS RVIALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 19      Ingr L2COS L2:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 20   Ingr L2COS L2ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 21     Ingr L2COS DEF:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 22      Ingr L3COS MD:     4/    32 (current/alloc) rules. Util. 
12.500%
   DB 23      Ingr L3COS L3:     0/   960 (current/alloc) rules. Util. 
0.000%
   DB 24   Ingr L3COS L3ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 25     Ingr L3COS DEF:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 26   Ingr L2COS MD-v6:     5/    20 (current/alloc) rules. Util. 
25.000%
   DB 27  Ingr L2COS RVI-v6:     0/   208 (current/alloc) rules. Util. 
0.000%
   DB 28   Ingr L2COS RA-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 29   Ingr L2COS L2-v6:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 30  Ingr L2COS L2A-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 31  Ingr L2COS DEF-v6:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 32   Ingr L3COS MD-v6:     4/    20 (current/alloc) rules. Util. 
20.000%
   DB 33   Ingr L3COS L3-v6:     0/   468 (current/alloc) rules. Util. 
0.000%
   DB 34 Ingr L3COS L3ALL-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 35  Ingr L3COS DEF-v6:     2/     4 (current/alloc) rules. Util. 
50.000%

PFEM2(vty)# show tcam vendor 6 instance 3 rules
Instance 3
   DB  0         Ingr L2COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  1          Ingr PACL:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB  2          Ingr VACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  3         Ingr L3COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  4          Ingr RACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  5           Egr PACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  6           Egr PCL1:     0/   508 (current/alloc) rules. Util. 
0.000%
   DB  7      Ingr L2COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB  8       Ingr PACL-v6:     0/   256 (current/alloc) rules. Util. 
0.000%
   DB  9       Ingr VACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 10        Egr PACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 11        Egr PCL1-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 12      Ingr L3COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 13      Ingr L3DYN-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 14       Ingr RACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 15         Ingr L2DYN:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 16      Ingr L2COS MD:     5/    28 (current/alloc) rules. Util. 
17.857%
   DB 17     Ingr L2COS RVI:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB 18  Ingr L2COS RVIALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 19      Ingr L2COS L2:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 20   Ingr L2COS L2ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 21     Ingr L2COS DEF:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 22      Ingr L3COS MD:     4/    32 (current/alloc) rules. Util. 
12.500%
   DB 23      Ingr L3COS L3:     0/   960 (current/alloc) rules. Util. 
0.000%
   DB 24   Ingr L3COS L3ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 25     Ingr L3COS DEF:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 26   Ingr L2COS MD-v6:     5/    20 (current/alloc) rules. Util. 
25.000%
   DB 27  Ingr L2COS RVI-v6:     0/   208 (current/alloc) rules. Util. 
0.000%
   DB 28   Ingr L2COS RA-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 29   Ingr L2COS L2-v6:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 30  Ingr L2COS L2A-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 31  Ingr L2COS DEF-v6:     2/     4 (current/alloc) rules. Util. 
50.000%
   DB 32   Ingr L3COS MD-v6:     4/    20 (current/alloc) rules. Util. 
20.000%
   DB 33   Ingr L3COS L3-v6:     0/   468 (current/alloc) rules. Util. 
0.000%
   DB 34 Ingr L3COS L3ALL-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 35  Ingr L3COS DEF-v6:     2/     4 (current/alloc) rules. Util. 
50.000%
PFEM2(vty)# show tcam vendor 6 instance 4 rules
Instance 4
   DB  0         Ingr L2COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  1          Ingr PACL:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB  2          Ingr VACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  3         Ingr L3COS:     0/  1024 (current/alloc) rules. Util. 
0.000%
   DB  4          Ingr RACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  5           Egr PACL:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB  6           Egr PCL1:     8/   508 (current/alloc) rules. Util. 
1.575%
   DB  7      Ingr L2COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB  8       Ingr PACL-v6:     0/   256 (current/alloc) rules. Util. 
0.000%
   DB  9       Ingr VACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 10        Egr PACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 11        Egr PCL1-v6:     7/   512 (current/alloc) rules. Util. 
1.367%
   DB 12      Ingr L3COS-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 13      Ingr L3DYN-v6:     0/   512 (current/alloc) rules. Util. 
0.000%
   DB 14       Ingr RACL-v6:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 15         Ingr L2DYN:     0/     0 (current/alloc) rules. Util. 
-1.000%
   DB 16      Ingr L2COS MD:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 17     Ingr L2COS RVI:     0/   696 (current/alloc) rules. Util. 
0.000%
   DB 18  Ingr L2COS RVIALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 19      Ingr L2COS L2:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 20   Ingr L2COS L2ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 21     Ingr L2COS DEF:     0/     4 (current/alloc) rules. Util. 
0.000%
   DB 22      Ingr L3COS MD:     0/    32 (current/alloc) rules. Util. 
0.000%
   DB 23      Ingr L3COS L3:     0/   960 (current/alloc) rules. Util. 
0.000%
   DB 24   Ingr L3COS L3ALL:     0/    28 (current/alloc) rules. Util. 
0.000%
   DB 25     Ingr L3COS DEF:     0/     4 (current/alloc) rules. Util. 
0.000%
   DB 26   Ingr L2COS MD-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 27  Ingr L2COS RVI-v6:     0/   208 (current/alloc) rules. Util. 
0.000%
   DB 28   Ingr L2COS RA-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 29   Ingr L2COS L2-v6:     0/   240 (current/alloc) rules. Util. 
0.000%
   DB 30  Ingr L2COS L2A-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 31  Ingr L2COS DEF-v6:     0/     4 (current/alloc) rules. Util. 
0.000%
   DB 32   Ingr L3COS MD-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 33   Ingr L3COS L3-v6:     0/   468 (current/alloc) rules. Util. 
0.000%
   DB 34 Ingr L3COS L3ALL-v6:     0/    20 (current/alloc) rules. Util. 
0.000%
   DB 35  Ingr L3COS DEF-v6:     0/     4 (current/alloc) rules. Util. 
0.000%

The other instances of vendor 6 are similar. Instances of vendor 7 give 
no information, showing only zeroes. The number shown are not giving me 
any clue.

Besides, if it actually is a TCAM full issue, why is it happening? The 
box with these cards should support 500K IPv4 routes, am I doing 
something wrong? Should I repartition the TCAM in some way, as I'm used 
to do with cisco boxes? I find no mention of this kind of configuration 
in the docs.

Thanks in advance for your kind replies.

			Bergonz


-- 
Ing. Michele Bergonzoni - Laboratori Guglielmo Marconi S.p.a.
Phone:+39-051-4392826 e-mail: bergonz at labs.it
alt.advanced.networks.design.configure.operate


More information about the juniper-nsp mailing list