[c-nsp] CoPP not catching software-switched CEF

[Blake Willis]

Greetings all,

 	During an attack towards a customer yesterday, it seems that on one 
router the attack traffic was punted and interrupt-switched in the 
software-switched CEF path (on 720/3BXL, SXD7b).  The destination was far far 
away from the router in question and the router was not the ingress or egress 
point in the network for the flow.  The flow (grabbed with software-switched 
netflow) was:

Single flow, IP proto 0, src port 0, dst port 0, TOS 0, tcp_flags 0x10 (ACK)

Flows  Packets    Bytes    pps     bps      bpp
     1  63.5 M     5.2 G    51799   33.2 M   84

 	The ingress router had no trouble forwarding this with its PFC, but the 
next one didn't (about the only difference between the two is that the input 
interface on the affected router is an L3 LACP channel on a 6516 and output is 
an enhanced OSM).  Netint forwarding is using about 30% cpu, and shows about 7k 
netint throttles/sec (mask is 1k/2k):

suffering-bxl#proc
CPU utilization for five seconds: 46%/31%; one minute: 48%; five minutes: 48%
suffering-bxl#sh msfc netint | incl count
  throttle count=231480922, timer count=98903862
suffering-bxl#sh msfc netint | incl count
  throttle count=231487564, timer count=98909697
suffering-bxl#sh msfc netint | incl count
  throttle count=231494050, timer count=98915276
suffering-bxl#proc
CPU utilization for five seconds: 40%/33%; one minute: 49%; five minutes: 48%

It isn't hitting an MLS rate limiter (otherwise it would be rate-limited and not eating cpu):

suffering-bxl#sh mls rate-limit usage | ex #
                              Rate Limiter Type     Packets/s   Burst
                            ---------------------   ---------   -----
                                    ICMP REDIRECT          10     255
                                      MTU FAILURE          10     255
                                  UCAST IP OPTION          10     255
                                      TTL FAILURE         500     255
                                        CEF GLEAN         500     255
                                   ACL BRIDGED IN          10     255
                                  ACL BRIDGED OUT          10     255
                                   IP RPF FAILURE          10     255
                            ICMP UNREAC. NO-ROUTE          10     255
                            ICMP UNREAC. ACL-DROP          10     255
                                        IP ERRORS          10     255
                                      IP FEATURES          10     255
                                      CAPTURE PKT        1000       1

So normally any punted traffic that doesn't hit an MLS RL is picked up by CoPP, right?

suffering-bxl#sh policy-map control-plane | incl rate
         5 minute offered rate 3104 bps
       5 minute offered rate 10000 bps, drop rate 0 bps
         5 minute offered rate 0 bps
       5 minute offered rate 0 bps, drop rate 0 bps
       5 minute offered rate 57000 bps, drop rate 0 bps

Nope.  BTW my CoPP looks like this:

ip access-list extended critical-protocols
  deny <bunch of core protocols>
  permit ip any any
class-map match-all copp-ip
   match access-group name critical-protocols
policy-map copp-policy
   class pingme
      police 256000 64000 64000 conform-action transmit exceed-action drop
   class copp-ip
      police 32000 8000 8000 conform-action transmit exceed-action drop
   class class-default	! don't police IS-IS

This traffic doesn't hit the CoPP ACL (although IP proto 0 is "ip any any"):

suffering-bxl#sh access-lists critical-protocols | incl permit
     380 permit ip any any (33882814 matches)
suffering-bxl#sh access-lists critical-protocols | incl permit
     380 permit ip any any (33882867 matches)
suffering-bxl#sh access-lists critical-protocols | incl permit
     380 permit ip any any (33882901 matches)

or show up on the EOBC:

suffering-bxl#sh eobc | incl rate
         rx rate (bits/sec) = 2340000    rx rate (packets/sec) = 222
         tx rate (bits/sec) = 112000     tx rate (packets/sec) = 216

(BTW, about the same rate on the SP side of the EOBC as well).

 	So, it seems that CoPP doesn't catch software-switched CEF (netint) 
traffic as it apparently isn't punted to the MSFC via the EOBC.  I suppose a 
workaround could be to apply a CoPP-like policy to every interface on the box 
using CAR ("rate-limit input access-group...") as that would likely catch it 
just as soft-switched netflow does, but that's fairly logistically annoying.

 	I won't have a chance to simulate this in the lab until after the 
holidays.  Any experiences or comments are welcome...

---
  Blake Willis
  Network Engineer
  blake at 2112 dot net