[c-nsp] One Cat6k/Sup2T is software switching, its identical partner is not

[Jeroen van Ingen]

Calling on your collective knowledge here, because our TAC case is 
progressing quite slowly. I hope someone is around who has in-depth 
knowledge of "feature manager" on Cat6k/Sup2T.

We have two Cat6k's with Sup2T in our network, both running IOS 
15.1(1)SY3. They function as distribution switches for several buildings 
and the configs are virtually identical, aside from IP addresses. Each 
building is connected to both boxes on a 10 GE interface on a 6904 
linecard; interfaces are in routed mode with a subinterface for each 
VLAN that we want to route and we run HSRP for gateway redundancy. I'll 
call them "router A" and "router B" from here.

During a maintenance window we rebooted router A. When it came back, it 
logged a few %FMCORE-4-RACL-REDUCED messages for subinterfaces and it 
was obviously forwarding part of the incoming traffic in software.

The thing is: on router B, all interfaces and subinterfaces are hardware 
switching ("not reduced" state) while on router A there's a clear 
pattern in subinterfaces in "partially reduced" state and subinterfaces 
in "not reduced" state. Again, identical config on both routers, 
verified by checking a vimdiff of "show running-config" and "show 
running-config all" between the routers.

We were able to create an extra subinterface on both routers that we can 
manipulate without affecting user traffic; on router A we can "toggle" 
that subinterface between "partially reduced" by removing PBR config and 
"not reduced" by adding PBR config. On router B, again, both with and 
without PBR on the subint it always switches in hardware.

With "debug fm features" and "debug fm core all" I checked the logging 
and on router A I see something interesting:

fm_core_notify_feat_exception_event() called for label: 51 i/f: 
TenGigabitEthernet2/15.1799 exception event: 0
-Traceback= 455BA44z 455CFE4z 45550D8z 4550E30z 45512B8z 45526B0z 
5250AE0z 524A374z

...and that's where I have no idea how to continue. Router A has these 
errors, router B doesn't. My gut feeling is that this would be the 
underlying cause for the "partially reduced" state and software 
switching of part of the traffic.

Anyone with ideas how to dig deeper?


Regards,

Jeroen van Ingen
ICT Service Centre
University of Twente, P.O.Box 217, 7500 AE Enschede, The Netherlands