[c-nsp] One Cat6k/Sup2T is software switching, its identical partner is not

[Jeroen van Ingen]

Update for anyone interested: it's been classified as a new bug, 
https://tools.cisco.com/bugsearch/bug/CSCuu15276. Cisco was able to 
reproduce it in lab and are currently investigating the root cause.


Regards,

Jeroen van Ingen
ICT Service Centre
University of Twente, P.O.Box 217, 7500 AE Enschede, The Netherlands


On 04/17/2015 01:16 PM, Jeroen van Ingen wrote:
> Calling on your collective knowledge here, because our TAC case is
> progressing quite slowly. I hope someone is around who has in-depth
> knowledge of "feature manager" on Cat6k/Sup2T.
>
> We have two Cat6k's with Sup2T in our network, both running IOS
> 15.1(1)SY3. They function as distribution switches for several buildings
> and the configs are virtually identical, aside from IP addresses. Each
> building is connected to both boxes on a 10 GE interface on a 6904
> linecard; interfaces are in routed mode with a subinterface for each
> VLAN that we want to route and we run HSRP for gateway redundancy. I'll
> call them "router A" and "router B" from here.
>
> During a maintenance window we rebooted router A. When it came back, it
> logged a few %FMCORE-4-RACL-REDUCED messages for subinterfaces and it
> was obviously forwarding part of the incoming traffic in software.
>
> The thing is: on router B, all interfaces and subinterfaces are hardware
> switching ("not reduced" state) while on router A there's a clear
> pattern in subinterfaces in "partially reduced" state and subinterfaces
> in "not reduced" state. Again, identical config on both routers,
> verified by checking a vimdiff of "show running-config" and "show
> running-config all" between the routers.
>
> We were able to create an extra subinterface on both routers that we can
> manipulate without affecting user traffic; on router A we can "toggle"
> that subinterface between "partially reduced" by removing PBR config and
> "not reduced" by adding PBR config. On router B, again, both with and
> without PBR on the subint it always switches in hardware.
>
> With "debug fm features" and "debug fm core all" I checked the logging
> and on router A I see something interesting:
>
> fm_core_notify_feat_exception_event() called for label: 51 i/f:
> TenGigabitEthernet2/15.1799 exception event: 0
> -Traceback= 455BA44z 455CFE4z 45550D8z 4550E30z 45512B8z 45526B0z
> 5250AE0z 524A374z
>
> ...and that's where I have no idea how to continue. Router A has these
> errors, router B doesn't. My gut feeling is that this would be the
> underlying cause for the "partially reduced" state and software
> switching of part of the traffic.
>
> Anyone with ideas how to dig deeper?
>
>
> Regards,
>
> Jeroen van Ingen
> ICT Service Centre
> University of Twente, P.O.Box 217, 7500 AE Enschede, The Netherlands
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/