[c-nsp] One Cat6k/Sup2T is software switching, its identical partner is not

[Jeroen van Ingen]

On 04/20/2015 05:39 PM, Matthew Huff wrote:
> Traffic destined to the switch rather than through the switch is
> generally software switched. Too much software switched traffic
> causing CPU issues could cause other traffic not to be installed in
> the ASIC causing a cascade failure.

I'm not sure that I understand what you're saying here. The box isn't 
busy at all right now because it's not the HSRP active for any connected 
network; it's doing some egress traffic for connected networks, but it's 
getting hardly any ingress because its partner router currently runs all 
gateway IPs. According to the "show fm" outputs the box is only impacted 
for ingress traffic on several subinterfaces.

CPU is pretty low:

Router-A#show proc cpu
CPU utilization for five seconds: 9%/1%; one minute: 10%; five minutes: 10%

If it were very busy, for example if I would let it claim all gateway 
IPs provided via HSRP, then I'd expect far more busy CPU (and going up & 
down with user traffic levels) and a certain probability that ARP 
resolving might fail now and then, resulting in longer standing glean 
adjacencies for egress traffic, which might indeed have a cascade-like 
effect.

> Also, since you are running HSRP, what do you have your
> "mac-address-table aging-time" set to on both boxes? You might be
> running into asymmetrical traffic flows causing unicast flooding.

All interfaces are in routed mode, ie "no switchport". Therefore the box 
doesn't do any MAC learning; it only does ARP resolution.

> Depending on your traffic levels, this could cause major issue. The
> recommended setting is around 14400, which is not the default. Google
> for either the command or 6500 unicast flooding for details.

Yes, I'm aware of unicast flooding and how it can impact a network, but 
in this specific topology I don't see how it could be related.

> During a maintenance window, I probably would suggest doing a
> complete power off, reseat the cards, and see if that makes any
> difference. I know of a few ASIC bugs that have been resolved over
> the years that are only fixed with either a linecard reset or a
> complete powerdown. The multicast ASIC bug we ran into on route
> topology changes was finally fixed in 12.2(33)SXI14 or later. Make
> sure you are running the latest linecard firmware release as well (at
> least 12.2(18r)S1)

Tomorrow 5 AM we have a maintenance window, so I'll probably try that 
anyway. Still doubt if it would work, because this issue has persisted 
even after replacing the supervisor, which has been done with a full 
power off. But we haven't reseated the line cards yet, so that's worth a 
try.

By the way, the WS-X6824-SFP card runs fw 12.2(18r)S1 and both 
WS-X6904-40G cards in the chassis run 12.2(50r)SYL. I re-checked and we 
see the issue not only on the two 6904 cards but also on the 6824 card.

Thanks for thinking with me.


Regards,

Jeroen van Ingen
ICT Service Centre
University of Twente, P.O.Box 217, 7500 AE Enschede, The Netherlands