[j-nsp] Cannot program filter pfe-cos-cl-631-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries

Chuck Anderson cra at fea.st
Wed Oct 12 14:36:02 EDT 2022


On Wed, Oct 12, 2022 at 08:40:46AM +0300, Saku Ytti wrote:
>   - show filter dram
>   - show filter hw X
>   - show filter hw X show_term_info
> 
> I lost a fight with JTAC about whether the TCAM exhausting filter
> should be a commit failure or not. Argument was along the line 'well
> you can keep adding routes even if you exhaust TCAM, so this should be
> the same'.
> I'm absolutely certain there are many QFX and EX networks out there
> with wildly different filters programmed than what they believe they
> have.

Switching platform (2199 Mhz Pentium processor, 511MB memory, 0KB flash)

FPC0(ex4300-48mp vty)# show filter dram
Name                   Bytes        Allocs         Frees      Failures
---------------------------------------------------------------------------
filter                  62940          1198           395          0
filter-halp                 0             0             0          0
---------------------------------------------------------------------------

Total DFW Dram Usage obtained from global handle:
Total DFW Dram Usage:   78680 bytes
Total DFW allocs:       740
Total DFW frees:        0
Outstanding DFW allocs: 740

Total DFW Dram Usage obtained from all filters:
Total DFW Dram Usage:   78704 bytes
Total DFW allocs:       740
Total DFW frees:        0
Outstanding DFW allocs: 740

FPC0(ex4300-48mp vty)# 
FPC0(ex4300-48mp vty)# show filter
Program Filters:
---------------
   Index     Dir     Cnt    Text     Bss  Name
--------  ------  ------  ------  ------  --------

Term Filters:
------------
   Index    Semantic    Name
--------  ----------------
       1  Classic   ROUTING-ENGINE
       2  Classic   ROUTING-ENGINE6
       3  Classic   ACCESS-FILTER
   17000  Classic   __default_arp_policer__
   57006  Classic   __jdhcpd__
   57007  Classic   __dhcpv6__
   65008  Classic   __jdhcpd_l2_snoop_filter__
16777216  Classic   fnp-filter-level-all
46137360  Classic   pfe-cos-cl-610-5-1
46137361  Classic   pfe-cos-cl-611-5-1
46137362  Classic   pfe-cos-cl-612-5-1
46137363  Classic   pfe-cos-cl-613-5-1
46137364  Classic   pfe-cos-cl-614-5-1
46137365  Classic   pfe-cos-cl-615-5-1
46137366  Classic   pfe-cos-cl-616-5-1
46137367  Classic   pfe-cos-cl-617-5-1
46137368  Classic   pfe-cos-cl-618-5-1
46137369  Classic   pfe-cos-cl-619-5-1
46137370  Classic   pfe-cos-cl-620-5-1
46137371  Classic   pfe-cos-cl-621-5-1
46137372  Classic   pfe-cos-cl-622-5-1
46137373  Classic   pfe-cos-cl-623-5-1
46137374  Classic   pfe-cos-cl-624-5-1
46137375  Classic   pfe-cos-cl-625-5-1
46137376  Classic   pfe-cos-cl-626-5-1
46137377  Classic   pfe-cos-cl-627-5-1
46137378  Classic   pfe-cos-cl-628-5-1
46137379  Classic   pfe-cos-cl-629-5-1
46137380  Classic   pfe-cos-cl-630-5-1
46137381  Classic   pfe-cos-cl-631-5-1
46137382  Classic   pfe-cos-cl-632-5-1
46137383  Classic   pfe-cos-cl-633-5-1
46137384  Classic   pfe-cos-cl-634-5-1
46137385  Classic   pfe-cos-cl-635-5-1
46137386  Classic   pfe-cos-cl-636-5-1
46137387  Classic   pfe-cos-cl-637-5-1
46137388  Classic   pfe-cos-cl-638-5-1
46137389  Classic   pfe-cos-cl-639-5-1
46137390  Classic   pfe-cos-cl-640-5-1
46137391  Classic   pfe-cos-cl-641-5-1
46137392  Classic   pfe-cos-cl-642-5-1
46137393  Classic   pfe-cos-cl-643-5-1
46137394  Classic   pfe-cos-cl-644-5-1
46137395  Classic   pfe-cos-cl-645-5-1
46137396  Classic   pfe-cos-cl-646-5-1
46137397  Classic   pfe-cos-cl-647-5-1
46137398  Classic   pfe-cos-cl-648-5-1
46137399  Classic   pfe-cos-cl-649-5-1
46137400  Classic   pfe-cos-cl-656-5-1
46137401  Classic   pfe-cos-cl-657-5-1
46137402  Classic   pfe-cos-cl-658-5-1
46137403  Classic   pfe-cos-cl-655-5-1
46137404  Classic   pfe-cos-cl-650-5-1
46137405  Classic   pfe-cos-cl-651-5-1
46137406  Classic   pfe-cos-cl-652-5-1
46137407  Classic   pfe-cos-cl-653-5-1
46137408  Classic   pfe-cos-cl-654-5-1
92274688  Classic   __jdhcpd_l2_dai_filter__
125829120  Classic   __jdhcpd_security_dhcpv6_l2_snoop_filter__
130023424  Classic   __jdhcpd_security_icmpv6_l2_snoop_filter__
142606336  Classic   __jdhcpd_l3_tag__
142606337  Classic   __dhcpv6_l3_tag__

Resolve Filters:
---------------
   Index
--------

FPC0(ex4300-48mp vty)#  show filter hw 46137374
======================
Filter index   : 46137374
======================

- Filter name  : pfe-cos-cl-624-5-1

+ Hardware Instance : 1
  + Hardware key (struct brcm_dfw_hw_key_t):
    - Type          : VFP_IL2L3_COS
    - Vlan id       : 0
    - Direction     : ingress
    - Protocol      : 35 (L2 Bridge)
    - Port class id : 0
    - Class id      : 0
    - Loopback      : 0
    - Port          : 0(xe-1)
    - Vlan tag      : 0
    - Non-overflow  : 1
  + FP usage info (struct brcm_dfw_fp_t):
    - Group                           : VFP IL2L3 COS group (143)
    - My Mac                          : 00:00:00:00:00:00
    - Loopback Reference Count        : 00000000
    - IFL Type                        : unknown (0)
    + List of tcam entries            : [ total: 0; ]
        - Pipe: 0; []
    + List of ranges                  : [ total: 0; ]
        - Pipe: 0 []
    + List of interface match entries : [ total: 0; ]
        - Pipe: 0 []
    + List of dot1q-tag match entries : [ total: 0; ]
        - Pipe: 0 []
    - List of l3 ifl index entries    : [ total: 0; ]
    + List of vfp tcam entries        : [ total: 0; ]
        - Pipe: 0 []
  + Misc info (struct brcm_dfw_misc_info_t):
    - List of <anlz_id, entry_id> : [ total: 0; ]
  + Bind point info (union brcm_dfw_bind_point_info_t):
    - No grouping possible
  + Programmed: NO
  + BD ID     : 500
  + Total TCAM entries available: 512
  + Total TCAM entries needed   : 1
  + Term Expansion:
    - Term    1: will expand to     1 term : Name "cos-cl-624-5-1"
  + Term TCAM entry requirements:
    - Term    1: needs     1 TCAM entry  : Name "cos-cl-624-5-1"
  + Total TCAM entries available: 512
  + Total TCAM entries needed   : 1

 Total hardware instances: 1

FPC0(ex4300-48mp vty)# 
FPC0(ex4300-48mp vty)#  show filter hw 46137374 show_term_info
======================
Filter index   : 46137374
======================

- Filter name  : pfe-cos-cl-624-5-1

+ Hardware Instance : 1
  + Hardware key (struct brcm_dfw_hw_key_t):
    - Type          : VFP_IL2L3_COS
    - Vlan id       : 0
    - Direction     : ingress
    - Protocol      : 35 (L2 Bridge)
    - Port class id : 0
    - Class id      : 0
    - Loopback      : 0
    - Port          : 0(xe-1)
    - Vlan tag      : 0
    - Non-overflow  : 1
  + FP usage info (struct brcm_dfw_fp_t):
    - Group                           : VFP IL2L3 COS group (143)
    - My Mac                          : 00:00:00:00:00:00
    - Loopback Reference Count        : 00000000
    - IFL Type                        : unknown (0)
    + List of tcam entries            : [ total: 0; ]
        - Pipe: 0; []
    + List of ranges                  : [ total: 0; ]
        - Pipe: 0 []
    + List of interface match entries : [ total: 0; ]
        - Pipe: 0 []
    + List of dot1q-tag match entries : [ total: 0; ]
        - Pipe: 0 []
    - List of l3 ifl index entries    : [ total: 0; ]
    + List of vfp tcam entries        : [ total: 0; ]
        - Pipe: 0 []
  + Misc info (struct brcm_dfw_misc_info_t):
    - List of <anlz_id, entry_id> : [ total: 0; ]
  + Bind point info (union brcm_dfw_bind_point_info_t):
    - No grouping possible
  + Programmed: NO
  + BD ID     : 500
  + Total TCAM entries available: 512
  + Total TCAM entries needed   : 1
  + Term Expansion:
    - Term    1: will expand to     1 term : Name "cos-cl-624-5-1"
  + Term TCAM entry requirements:
    - Term    1: needs     1 TCAM entry  : Name "cos-cl-624-5-1"
  + Total TCAM entries available: 512
  + Total TCAM entries needed   : 1

 Total hardware instances: 1



FPC0(ex4300-48mp vty)# show filter hw groups
Unit:0 Group Information:
> VFP groups:
    VFP group for DYN IPSG group id: 231. Pipe:  0 Entries:  238 Max Entries(total_available):  256(1024) Pri:  3 Slice:  2 Def Entries:  0
> IFP groups:
                     iRACL group id: 33. Pipe:  0 Entries: 1148 Max Entries(total_available): 2048(2560) Pri:  6 Slice:  4 Def Entries:  0
                     iPACL group id: 25. Pipe:  0 Entries:   12 Max Entries(total_available):  512(1024) Pri:  8 Slice:  2 Def Entries:  0
                   Dynamic group id: 17. Pipe:  0 Entries:  141 Max Entries(total_available):  512(1024) Pri:  5 Slice:  2 Def Entries:  0
             Dynamic HiGig group id: 37. Pipe:  0 Entries:   46 Max Entries(total_available):  512(2048) Pri:  4 Slice:  1 Def Entries:  0
> EFP groups:

FPC0(ex4300-48mp vty)# show filter summary
Filter Summary:
Number of Program  Filters = 0
Number of Term     Filters = 62
Number of Resolve  Filters = 0
Number of Simple  Filters = 0
Number of Implicit Filters = 58
Number of Other    Filters = 0
Number of cached non-local Filters = 0
Number of post cli Implicit Filters = 0
Number of service filter templates = 0

Total Number of Filters = 62

Protocol Filters:
Number of PROTO_IP Filters = 4
Number of PROTO_IP6 Filters = 3
Number of PROTO_MPLS Filters = 0
Number of PROTO_VPLS Filters = 55
Number of PROTO_CCC Filters = 0
Number of PROTO_ANY Filters = 0
Number of PROTO_BRIDGE Filters = 0
Number of Filters with unknown protocols = 0

Application Filters:
Number of Implicit Filters from OAM = 0
Number of Implicit Filters from JDHCPD = 2
Number of Implicit Filters from L2CPD = 1
Number of Implicit Filters from DHCP L2 = 1
Number of Implicit Filters from PGCPD_COMPILED = 0
Number of Implicit Filters from PGCPD_FUF = 0
Number of Implicit Filters from FTAPLITE = 0
Number of Implicit Filters from SA_VAL = 0
Number of Implicit Filters from JDHCPD_L2_FUF = 0
Number of Implicit Filters from SDK1 = 0
Number of Implicit Filters from SDK2 = 0
Number of Implicit Filters from SDK3 = 0
Number of Implicit Filters from SDK4 = 0
Number of Implicit Filters from SDK5 = 0
Number of Implicit Filters from TEST1 = 0
Number of Implicit Filters from TEST2 = 0
Number of Implicit Filters from DOT1XD = 0
Number of Implicit Filters from DYNAMIC SERVICES = 0
Number of Implicit Filters from LPDFD = 0
Number of Implicit Filters from Propagated Config = 0
Number of Implicit Filters from RPD = 0
Number of Implicit Filters from unknown = 0
Number of Implicit Filters from unknown = 49
Number of Implicit Filters from LI_DFCD = 0
Number of Implicit Filters from ESWD = 0
Number of Implicit Filters from ESWD EVB = 0
Number of Implicit Filters from PPMD = 0
Number of Implicit Filters from SPD = 0
Number of Implicit Filters from RMOPD = 0
Number of Implicit Filters from unknown = 0
Number of Implicit Filters from UACD = 0
Number of Implicit Filters from MCOS = 0
Number of Implicit Filters from RMPSD = 0
Number of Implicit Filters from DHCP L2 DAI = 1
Number of Implicit Filters from OPENFLOWD = 0
Number of Implicit Filters from VMOND = 0
Number of Implicit Filters from DHCP IPV6 = 1
Number of Implicit Filters from DHCP ICMPV6 = 1
Number of Implicit Filters from JDHCPD L3 TAG = 2
Number of Implicit Filters from BBE1 = 0
Number of Implicit Filters from BBE1_LI = 0
Number of Implicit Filters from BBE1_BGF = 0
Number of Implicit Filters from URL-FILTERD = 0
Number of Implicit Filters from URL 2 FILTERD = 0
Number of Implicit Filters from TWAMP = 0
Number of Implicit Filters from CAED-GLOBAL = 0
Number of Implicit Filters from IDS_FILTER = 0
Number of Implicit Filters from RTLOGD = 0
Number of Implicit Filters from CAED-LOCAL = 0


FPC0(ex4300-48mp vty)# show filter statistcs
Filter Manager Statistics:
          0 program filter add
          0 program filter delete
          0 program filter change
          0 filter log get
       2291 filter counter get
          0 filter counter reset
          0 filter log clear
          0 filter dcu stats get
          0 filter scu stats get
          0 rpf stats get
          0 prefix action stats get
          0 prefix action stats clear
        385 filter apply
          2 policer
          0 three-color policer
          0 hierarchical policer
          0 global msg
          0 prefix specifc action
          0 timer
          0 notify owner id
         13 filter head
         47 filter term
         14 filter state
          0 multicast statistics support

Filter Manager Errors:
          0 program filter add
          0 program filter delete
          0 program filter change
          0 filter log get
          0 filter counter get
          0 filter counter reset
          0 filter log clear
          0 filter dcu stats get
          0 filter scu stats get
          0 rpf stats get
          0 prefix action stats get
          0 prefix action stats clear
          0 filter apply
          0 policer
          0 three-color policer
          0 hierarchical policer
          0 global msg
          0 prefix specifc action
          0 timer
          0 notify owner id
          0 filter head
          0 filter term
          0 filter state
          0 multicast statistics support

More details extracted from the full "show filter hw all" output:

======================
Filter index   : 46137374
======================

- Filter name  : pfe-cos-cl-624-5-1
Filter index = 46137374
Optimization flag: 0x0
Filter notify host id = 0
Pfe Mask = 0x0
jnh inst = 0x0
Filter properties: None
Filter state = CONSISTENT
term cos-cl-624-5-1
term priority 0
    ethertype  
         33024 

    then
	accept

+ Hardware Instance : 1
  + Hardware key (struct brcm_dfw_hw_key_t):
    - Type          : VFP_IL2L3_COS
    - Vlan id       : 0
    - Direction     : ingress
    - Protocol      : 35 (L2 Bridge)
    - Port class id : 0
    - Class id      : 0
    - Loopback      : 0
    - Port          : 0(xe-1)
    - Vlan tag      : 0
    - Non-overflow  : 1
  + FP usage info (struct brcm_dfw_fp_t):
    - Group                           : VFP IL2L3 COS group (143)
    - My Mac                          : 00:00:00:00:00:00
    - Loopback Reference Count        : 00000000
    - IFL Type                        : unknown (0)
    + List of tcam entries            : [ total: 0; ]
        - Pipe: 0; []
    + List of ranges                  : [ total: 0; ]
        - Pipe: 0 []
    + List of interface match entries : [ total: 0; ]
        - Pipe: 0 []
    + List of dot1q-tag match entries : [ total: 0; ]
        - Pipe: 0 []
    - List of l3 ifl index entries    : [ total: 0; ]
    + List of vfp tcam entries        : [ total: 0; ]
        - Pipe: 0 []
  + Misc info (struct brcm_dfw_misc_info_t):
    - List of <anlz_id, entry_id> : [ total: 0; ]
  + Bind point info (union brcm_dfw_bind_point_info_t):
    - No grouping possible
  + Programmed: NO
  + BD ID     : 500
  + Total TCAM entries available: 512
  + Total TCAM entries needed   : 1
  + Term Expansion:
    - Term    1: will expand to     1 term : Name "cos-cl-624-5-1"
  + Term TCAM entry requirements:
    - Term    1: needs     1 TCAM entry  : Name "cos-cl-624-5-1"
  + Total TCAM entries available: 512
  + Total TCAM entries needed   : 1

 Total hardware instances: 1
======================


> On Wed, 12 Oct 2022 at 05:33, Chuck Anderson via juniper-nsp
> <juniper-nsp at puck.nether.net> wrote:
> >
> > Has anyone seen these errors and know what the cause is?
> >
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-624-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-626-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-631-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-631-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-632-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-632-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-633-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-633-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-634-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-634-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-638-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-638-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-647-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-647-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-656-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-656-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-657-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-657-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-655-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-652-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-652-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-653-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-653-5-1" is NOT programmed in HW
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Cannot program filter pfe-cos-cl-654-5-1 (type VFP_IL2L3_COS) -TCAM has 0 free entries
> > Oct 11 21:41:02  ex4300-48mp fpc0 DFWE ERROR DFW: Filter : "pfe-cos-cl-654-5-1" is NOT programmed in HW
> >
> > There is plenty of TCAM space for IRACL/IPACL entries, so this seems to be some issue with a different TCAM partition?
> >
> > ex4300-48mp> show pfe filter hw summary
> >
> > Slot 0
> >
> > Unit:0:
> > Group                    Group-ID       Allocated      Used           Free
> > ---------------------------------------------------------------------------
> > > Ingress filter groups:
> >   iRACL group            33             2048           1148           900
> >   iPACL group            25             512            12             500
> > > Egress filter groups:
> >
> > Slot 1
> >
> > Unit:0:
> > Group                    Group-ID       Allocated      Used           Free
> > ---------------------------------------------------------------------------
> > > Ingress filter groups:
> >   iRACL group            33             2048           1148           900
> >   iPACL group            25             512            12             500
> > > Egress filter groups:


More information about the juniper-nsp mailing list