[c-nsp] cisco-nsp Digest, Vol 58, Issue 20

[mack]

> Message: 9
> Date: Thu, 6 Sep 2007 09:21:14 -0500
> From: "Andrew Milner" <andyman33 at gmail.com>
> Subject: [c-nsp] PBR performance gotchas
> To: cisco-nsp at puck.nether.net
> Message-ID:
>         <c37774e90709060721i660f2097o52b4e2e245b6bfb at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Greetings
>
> We're contemplating a design that would make extensive use of policy
> based routing on our 7600 RSP720/WS-6704's..  Back in the day, you'd
> never do something like that because of performance but now it seems
> like that is fixed?  I don't have the means to test in a lab
> environment, so I'm just wondering if anyone has any experience with
> this and might be able to help me determine if/when this will fall
> over.
>
> Thanks in advance for any help,
>
> Andrew
>

First YMMV.
Second you should become intimately familiar with TCAM and the merge algorithm.

PBR interacts badly with other features that rely on TCAM (NAT, NBAR, Access Lists, DHCP Snooping, etc).
Only two can be configured without TCAM doing a merge or software switching.
This tends to lead to a TCAM explosion.
A merge also occurs if there is more than one "match ip address" rule in a policy.
Adding a second clause to a policy will generally result in a merge.
Adding a single additional ACL rule can easily multiply TCAM usage by a factor of 2.
Performance is unaffected by the number of ACL rules UNTIL you exceed available TCAM then traffic becomes software switched. If your bandwidth is high (which 6704s usual are), your CPU overloads, your router crashes, game over.

It is strongly suggested that high bandwidth/critical interfaces be configured with: "tcam priority high"
"show platform hardware capacity acl" can be used to measure tcam usage.
If a tcam rule is software switched it won't show up in usage.
"show platform software tcam counts" gives a global software view.
"show platform software tcam interface XXXXXX acl [in|out] ip" will show what is being inserted into the TCAM.
You can also show the actual TCAM VMR entries with "show fm pbr int XXXXX" if necessary (and sometimes it is).
"show fm summary" will let you know if something is software switched (ie. INACTIVE).
If tcam overflows and software switching starts a message will be logged (not sure on the logging level).

Other Caveats:
Recursive next-hop is not currently (12.2(18)SXF10) supported in HW so you must use a real interface.
12.2(33)SRB1 may have this resolved.

Routing to local destinations should be specifically excluded in the ACLs as PBR routes everything to the next-hop address (using default next-hop avoids that issue but probably doesn't do what you want).

----

As long as your configuration does not exhaust your TCAM you will be OK.
Understand the effects of your policies before you apply them to a live interface.
A test vlan with one server and traceroute can make a very good companion.
Espcially if the test bed has "tcam priority low" as this will clue you in to tcam overflow without effecting production traffic.

Hope this answers most of your questions.

LR Mack McBride
Network Administrator
Alpha Red, Inc.