[c-nsp] sup720 RP CPU utilisation with >20k adjacencies / IPv6 ND load?

[Dale W. Carder]

Hey Phil,



Thus spake Phil Mayers (p.mayers at imperial.ac.uk) on Thu, May 31, 2012 at 03:01:52PM +0100:
> All,
> 
> We route our edge networks on 6500s with a pretty high density of 1G
> ports to edge switches.
> 
> In the last week or so, we've seen a spike in RP CPU utilisation.
> This has coincided with AAAA records being installed on Facebook and
> some of our internal services, in preparation for world IPv6 rollout
> on Jun 6.
> 
> Effectively, although all our edge networks were IPv6-enabled, few
> clients lived in the neighbour table because there was little IPv6
> traffic; this has now changed, and from what I can see, most of the
> CPU is going on neighbour table & IPv4/ARP table maintenance. On a
> typical router:
> 
> CPU utilization for five seconds: 71%/15%; one minute: 71%; five
> minutes: 70%
> 
> ...and:
> 
>   5Sec   1Min   5Min TTY Process
> 12.15% 12.51% 12.37%   0 IPv6 ND
> 10.71% 11.07% 10.99%   0 ARP Input
>  5.51%  6.57%  6.51%   0 IPv6 Input
>  3.51%  3.29%  3.33%   0 CEF: IPv4 proces
>  3.03%  2.93%  2.92%   0 IP Input
>  2.95%  2.89%  2.84%   0 Earl NDE Task
> 
> A typical SVI config looks like this:
> 
> interface Vlan202
>  vrf forwarding PROD
>  ip address 192.168.202.254 255.255.255.0
>  ip verify unicast source reachable-via rx
>  no ip proxy-arp
>  ip flow ingress
>  standby version 2
>  standby 0 ip 192.168.202.1
>  standby 1 ipv6 autoconfig
>  ipv6 nd prefix 2001:db8:1:100::/64 900 600
>  ipv6 nd router-preference High
>  ipv6 traffic-filter IPV6_EDGE_NET_IN in
>  arp timeout 1200
> 
> Note that we are *not* using "ipv6 address", but rather specifying
> the nd prefix only; since we would want to set the timers in any
> event, we figured why bother with the address (we don't care about
> it for debugging or static hosts - these are edge networks, with
> everything using SLAAC).
> 
> The box has a fair number of adjacencies:
> 
> #sh mls cef adjacency usage
> 
> Adjacency Table Size:     1048576
> ACL region usage:         3
> Non-stats region usage:   132
> Stats region usage:       26881
> Total adjacency usage:    27016
> 
> ...and we see the CPU utilisation roughly track the number of adjacencies.
> 
> My question is: is there anything we can tweak to reduce the amount
> of CPU time spend in IPv6 ND (and maybe IPv4 ARP) maintenance?
> Obviously we can increase the arp timeout on IPv4 - is there an
> equivalent for IPv6? How does IOS behave w.r.t. ND table maintenance
> - when does it send NS messages to refresh the cache?

Our network and some of our peers have run into the same issues as we
did our v6 rollouts.  Try this out:

 ipv6 nd reachable-time 900000
 ipv6 nd ns-interval 5000

As for ipv6 addressing for routers, TIMTOWTDI, just like programming in
PERL :-).  We are on the other end of the spectrum, with every router SVI
assigned to be fe80::1.


interface Vlan42
 description The Vlan that is the Answer
 ip address 10.92.67.3 255.255.255.0
 ip verify unicast source reachable-via rx allow-self-ping
 ip helper-address 10.92.254.252
 no ip proxy-arp
 ip flow ingress
 ip pim dr-priority 4294967294
 ip pim sparse-mode
 ip multicast boundary G-T-LanMulticastBlock
 ip igmp access-group G-T-LanMulticastBlock
 ipv6 address FE80::3 link-local
 ipv6 address 2607:F388:E:100::3/64
 ipv6 nd reachable-time 900000
 ipv6 nd ns-interval 5000
 ipv6 nd other-config-flag
 ipv6 nd router-preference High
 ipv6 pim dr-priority 4294967295
 ipv6 dhcp relay destination 2607:F388::68:1
 ipv6 ospf 1 area 0
 standby version 2
 standby 0 ip 10.92.67.1
 standby 0 preempt
 standby 0 authentication vlan42
 standby 1 ipv6 FE80::1
 standby 1 preempt
 standby 1 authentication vlan42


Cheers,
Dale