[c-nsp] All multicast punting to CPU on 6500

[Robert Williams]

Hi all,

I’ve spent a while on this issue for a client now and can’t seem to track it down so I’m hoping someone will have a quick pointer for me. The problem has now been replicated in a controlled environment as follows.

        Platform: 6500/SUP-720
        Topology: Host <-> Unmanaged switch <-> 6500

The host is generating packets which are addressed to (any) of the multicast MAC addresses.

For example, but not limited to:

        03:bf:0a:00:05:d1
        01:00:5e:7f:05:77

The unmanaged switch floods these packets to all ports, because no devices on the LAN respond to those MACs and nobody belongs to any multicast groups like that.

This includes forwarding to the 6500 on an interface in which it has a presence itself (as a layer 3 gateway).

The CPU usage on the 6500 starts rising rapidly.

After doing an rspan capture of the RP/SP traffic on the control-plane it becomes apparent that every single packet of the stream (around 30k pps / 30mbits ) is hitting the control-plane.

In order to replicate this, we have a basic linux VM host directly attached to a 6500 here. We have then issued the following commands:

        # arp -s 10.1.2.3 03:bf:0a:12:34:56
        # hping3 -i u10 10.1.2.3

The 6500 has a pretty substantial CoPP on it, matching every required protocol separately and ending with a catch-all (match ip any any) which denies anything else. None of the counters on the CoPP are incrementing for this traffic, which I believe is due to it being multicast and thus not supported by CoPP.

So, my question / problem is - why is all this traffic hitting the CPU? We've compared rates and every single packet is making its way to the CPU, so no filtering is occurring at all. Based on the rate of CPU increase we believe it will reach 100% load around 250k PPS, give or take. Since the chassis has several 10GB cards in it, there is more than enough potential for this to occur even with very stringent multicast storm-control.

Some relevant config extracts:

        ip icmp rate-limit unreachable 10
        ip icmp rate-limit unreachable DF 10
        mls rate-limit multicast ipv4 fib-miss 100 100
        mls rate-limit multicast ipv4 connected 100 100
        mls rate-limit multicast ipv4 igmp 100 100
        mls rate-limit multicast ipv4 partial 100 100
        mls rate-limit multicast ipv6 default-drop 100 20
        mls rate-limit multicast ipv6 route-cntl share default-drop
        mls rate-limit unicast cef glean 1000 100
        no mls rate-limit unicast acl vacl-log
        mls rate-limit unicast ip rpf-failure 10 100
        mls rate-limit unicast ip icmp redirect 10 100
        mls rate-limit unicast ip icmp unreachable no-route 10 100
        mls rate-limit unicast ip icmp unreachable acl-drop 10 100
        mls rate-limit unicast ip errors 10 100
        mls rate-limit all ttl-failure 10 100
        mls rate-limit all mtu-failure 10 100
        ip igmp snooping source-only-learning limit 1000
        (no) ip igmp snooping ** I've tried with snooping both on and off, no impact.

So any ideas on why this is occurring, or how to stop it / limit it?

Thanks for your time as always!


Robert Williams
Custodian Data Centre
Email: Robert at CustodianDC.com
http://www.CustodianDC.com


Robert Williams
Backline / Operations Team
Custodian DataCentre
tel: +44 (0)1622 230382
email: Robert at CustodianDC.com
http://www.custodiandc.com/disclaimer.txt