[c-nsp] Operational impact of switching from ingress to egress replication mode

[Phil Mayers]

On 09/21/2010 09:58 PM, John Neiberger wrote:
> Sorry.  I also meant to say it's a Sup 720-3BXL. Based on what I can
> see on CCO, that thing can forward 400 Mpps of ipv4 traffic. Does that
> mean that I can set a rate limit of, say, 300 Mpps and somewhat guard
> the CPU from meltdown for a few moments?

I wish! 300Mpps hitting the CPU of a sup720 will kill it stone dead. It 
has (two) 600MHz CPUs, and they will not survive such a load.

There's lots of info in this in the archives, but in brief: the 
sup720/pfc3 architecture forwards most packets in hardware. Some packets 
are however punted to CPU; these include:

  1. Packets which need ARP resolution ("glean")

  2. Multicast packets, which are trickled to the CPU so the CPU can see 
them and build (and refresh) hardware forwarding state

  3. ACL and uRPF denies, which are trickled to the CPU so it can 
maintain counters

  4. Various other traffic like TTL failures, needing ICMP

  5. Obviously, packets address to the CPU (routing traffic, layer 2 PDUs)

Because the CPUs on these boxes are very, very puny, you want to limit 
what hits the CPU. There are two methods available:

  1. The "mls rate-limit" commands; these will place a simple numeric 
rate cap on certain types of traffic, and is done in hardware. There's 
no prioritisation, but for certain types of traffic (e.g. TTL failures) 
you can and should IMHO set low-ish limits. You SHOULD NOT use the 
"general" CEF limiter; because you should use...

  2. CoPP - basically QoS on traffic punted to CPU. This is superior 
because you can write ACLs defining what is most important to you, with 
very granular control over policy. It suffers a couple of problems - 
broadcast/multicast and non-IP traffic are done in software, and it 
can't distinguish between glean and receive traffic, making a 
default-deny policy tricky.


In short, common advice seems to be:

  1. Set low limits on the mls limiters for TTL & MTU failure, and 
optionally ACL-drop ICMP unreach:

mls rate-limit unicast ip icmp unreachable acl-drop 0
mls rate-limit all ttl-failure 100 10
mls rate-limit all mtu-failure 100 10

  2. Use CoPP for everything else; DO NOT use the glean or cef receive 
limiter

Search the archives and the Cisco docs for more info; it's not something 
I can summarise in 5 minutes I'm afraid.

In your case, if you are going to perform a task which will potentially 
punt a lot of multicast traffic to the CPU, I was suggesting that there 
are MLS limiters which will reduce this; see my earlier email; though we 
run with the defaults, which are (quite) high PPS values!

sh mls rate-limit  | inc MC
          MCAST NON RPF   Off                  -       -     -
         MCAST DFLT ADJ   On              100000     100  Not sharing
       MCAST DIRECT CON   Off                  -       -     -
       MCAST PARTIAL SC   On              100000     100  Not sharing