[c-nsp] TCAM debugging fun on 3560X

[Elmar K. Bins]

Hi guys,

I inherited a set of 3560X a while ago, and they are being used
for L3, v4 and v6, and - worst of all - filtering.

I'm constantly hitting the deck with the ACLs:

#sh platform tcam util

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

[...]

 IPv4 security aces:                          1024/1024        980/980



I'm also seeing:

#sh platform acl oacltcamfull

r1.rb#sh plat acl oacltcamfull
      Vlan       oacl_tcam_full_bitmap           notify_apps
        10                0x  0                     NOT-FULL
        12                0x  0                     NOT-FULL
        13                0x  0                     NOT-FULL
       100                0x  0                     NOT-FULL
       113                0x  0                     NOT-FULL
        14                0x  0                     NOT-FULL
        15                0x  0                     NOT-FULL
        16                0x  0                     NOT-FULL
        17                0x  0                     NOT-FULL
        18                0x  0                     NOT-FULL
        19                0x  0                     NOT-FULL
       119                0x  0                     NOT-FULL
        20                0x  1                         FULL
        21                0x  0                     NOT-FULL
       121                0x  0                     NOT-FULL
      Vlan    ipv6_oacl_tcam_full_bitmap         notify_apps

I could find no documentation at all about what the notify_apps
column means. Some weird documentation mentions "FULL" as
"fully deployed on the ASIC" as opposed to...err...?


FIRST QUESTION:

  So what does "FULL" and "NOT-FULL" mean, and what does it mean if
  a VLAN is *not* listed?



I regularly - especially when removing and reapplying ACLs, sometimes
while removing single rules - the dreaded "TCAM full, forwarding in software"
notice.


SECOND QUESTION:

How can I recover from software forwarding without rebooting the box?



Oh, and speaking of ACLs, it can get very interesting sometimes.
Just this morning I *removed* an entire ACL. After that,
"show platform tcam util" showed *more* usage in the IPv4 security aces line.

Most of the time, when I remove single entries from ACLs, the counter does not
change at all.

This all feels quite weird to me, but it could be that now another ACL is
committed to TCAM which didn't fit before, and this one is bigger.

THIRD QUESTION:

How can I find out which ACL uses how many slots in the TCAM, and which
ACLs are not committed to the TCAM? (Maybe this relates to Question 1)...




TCAM debugging is kind of weird, especially because of the lack of
documentation. If anyone here could shed some light on the subject,
I'd be very happy.

Yours,
	Elmar.


PS: Bonus question - what's the most cost effective way - apart from
    a Linux box - of adding a router-on-a-stick to the setup, means:
    which platform could one use? Hardware forwarding highly
    appreciated, of course, but a throughput of around 500kpps would
    be ok, too. Does not need to be Cisco. Should be able to handle
    a couple thousand ACL entries...