[c-nsp] c6k msfc - hsrp flapping

[lee.e.rian at census.gov]

Hi Mike,

"Michael Davis" <michael at michael-davis.com> wrote on 05/18/2006 04:34:02
PM:

> Hi Lee,
>
> CPU at 49% seems significantly high for a distributed system.

But doesn't it seem way too low to account for hsrp and eigrp flapping?
The other times we've had hsrp and eigrp flapping the cpu busy was over 90%
and there were lots of input queue drops.

>  Most
> of the packet forwarding ought to be going on in hardware and not
> affecting that number. I've seen situations where an 15mbps of
> unfragmented traffic (that needed fragmentation to egress the
> switch) took a 7600 cpu utilization from 0-1% to 40%.
>
> The fact that IPC counters are incrementing suggests that the Cat's
> line cards are having trouble talking to the MSFC or visa versa.

IPC?  I was looking at IBC counters.
  OK, I'm back.  tried all the options shown for "sh ipc ?" and nothing
looks bad there.

I saw something in the release notes about IBC being "... the internal
electrical interface between the switch processor and the route processor".
But I haven't found a description of the counters.

FX1000 resets sounds bad... but who know if an FX 1000 reset interrupts
traffic between the SP and RP or not?

> EIGRP and HSRP both generate multicast traffic.  Multicast traffic
> must be process switched, as do HSRP and EIGRP control plane
> functions. All of which are certainly contributing to your CPU
> levels.  High CPU levels cause IPC errors. IPC errors can result in
> HSRP, EIGRP or other control plane failures, input queue drops, etc
> 'cause the line card is waiting for the cpu to tell it how to switch
> the packet.

And I'm hoping there are some commands that would pinpoint where the
problem is occuring.  Or at least what's going wrong..  cpu busy and input
queue drops don't seem anywhere near high enough to explain hsrp and eigrp
flapping.

> Long and short, if you have a pair of 6500/7600s aggregating many
> vlans, with hsrp and eigrp adjacencies between all of them, you may
> be asking for trouble.

Eigrp adjacencies are already limited to 3 on each msfc.  But it's possible
we've got twice as many hsrp adjacencies as necessary - there's two hsrp
groups on most of the user vlan interfaces.  The idea was to use both
uplinks for traffic from hosts on the access layer switch by setting the
default gateway via dhcp according to the mac address being even or odd.  I
don't know if it's actually been done on more than a handful of vlans tho.

> High CPU:
>
http://www.cisco.com/en/US/partner/products/hw/routers/ps359/products_tech_note09186a00801c2af0.shtml#causes

"Software bug" seems to be the best match ;^)


> Campus design:
>
>
http://www.cisco.com/application/vnd.ms-powerpoint/en/us/guest/netsol/ns432/c649/cdccont_0900aecd802e9b1a.ppt

Nice presentation!  I did a not so quick read and we're following most of
the recommendations.

Thanks,
Lee