[c-nsp] 3750G-48 / 12.2(58)SE1 issue (word of warning)

[Jeff Kell]

We had a "need" (bugfix) to upgrade our 12.2(55)SE1 stack of 4 *
3750G-48s to 12.2(58)SE1 (IP Services).  The maintenance window was last
night, and appeared to go well, until the phone lit up...

To make a very long story as short as possible, to the best extent I can
figure it out...

The stack is running several VRFs, incoming traffic from the core
arriving on a 4-port cross-stack etherchannel trunk carrying multiple
vlans (one per VRF), with connectivity to the edge (default) out of the
core to our border routers.  All traffic (in-house as well as default)
traverses the same trunks.  IGP on the VRFs is EIGRP in all cases.

The borders lit up after the upgrade hitting reverse-path verify on our
ASAs.  Two of the VRFs were showing traffic from "other" VRFs where they
should never be seen.

It appears that ingress traffic on one vlan was landing on another
(and/or else egress traffic on one vlan was landing on another).  The
"crossed-plumbing" was exactly two pairs out of a possible 12, with
improper egress resulting on 2 destination vlans.  Everything else was fine.

If I shutdown the SVIs of the two problem vlans, everything worked fine
again (the "other two" vlans of the pair then routed correctly).

I checked default routes, CEF adjacencies, everything I could think
could be affecting things, but if either of the problem SVIs was brought
back up, the mess ensued again.

Rebooting the stack resulted in no change (well, actually, only one pair
was crossed initially, the second pair appeared after the second reboot).

Reloaded 12.2(55)SE1, brought the two SVIs back up, all was fine.

Still scratching my head.  I also applied 12.2(58) to a 3750E and 3750X
stack during the same window, no issues, very similar configuration and
adjacencies.

So for the 3750G platform, you might hold off, or at least cautiously
approach this update.  I have a TAC case on this one but it may be a
tough one to reproduce (I certainly don't want to repeat it here!).

The only "anomaly" during the update was the appearance of the following
errors during the reload (same 4 appeared in each of the 4 member's
startup logs):

> *Mar  1 00:04:14.938: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been
> ADDED to the stack
> *Mar  1 00:04:14.938: %STACKMGR-4-SWITCH_ADDED: Switch 2 has been
> ADDED to the stack
> *Mar  1 00:04:14.938: %STACKMGR-4-SWITCH_ADDED: Switch 3 has been
> ADDED to the stack
> *Mar  1 00:04:14.938: %STACKMGR-4-SWITCH_ADDED: Switch 4 has been
> ADDED to the stack
> *Mar  1 00:04:15.248: platform assert failure: hwidb->snmp_if_index ==
> ifIndex: ../src-hulc/src-common/hpm_idbman.c: 1843:
> hpm_register_idb_with_snmp
> *Mar  1 00:04:15.248: -Traceback= 229B110z 16AEC38z 184DBA8z 184F040z
> 184E298z 184D340z 1864174z 1832F7Cz 16B6824z 3112CECz 3112D90z
> 3112EF8z 22A6F50z 22A721Cz 1CC0FA8z 1CBAE1Cz
> *Mar  1 00:04:15.340: platform assert failure: hwidb->snmp_if_index ==
> ifIndex: ../src-hulc/src-common/hpm_idbman.c: 1843:
> hpm_register_idb_with_snmp
> *Mar  1 00:04:15.340: -Traceback= 229B110z 16AEC38z 184DBA8z 184F040z
> 184E298z 184D340z 1864174z 1832F7Cz 16B6824z 3112CECz 3112D90z
> 3112EF8z 22A6F50z 22A721Cz 1CC0FA8z 1CBAE1Cz
> *Mar  1 00:04:15.407: platform assert failure: hwidb->snmp_if_index ==
> ifIndex: ../src-hulc/src-common/hpm_idbman.c: 1843:
> hpm_register_idb_with_snmp
> *Mar  1 00:04:15.416: -Traceback= 229B110z 16AEC38z 184DBA8z 184F040z
> 184E298z 184D340z 1864174z 1832F7Cz 16B6824z 3112CECz 3112D90z
> 3112EF8z 22A6F50z 22A721Cz 1CC0FA8z 1CBAE1Cz
> *Mar  1 00:04:15.491: platform assert failure: hwidb->snmp_if_index ==
> ifIndex: ../src-hulc/src-common/hpm_idbman.c: 1843:
> hpm_register_idb_with_snmp
> *Mar  1 00:04:15.491: -Traceback= 229B110z 16AEC38z 184DBA8z 184F040z
> 184E298z 184D340z 1864174z 1832F7Cz 16B6824z 3112CECz 3112D90z
> 3112EF8z 22A6F50z 22A721Cz 1CC0FA8z 1CBAE1Cz

I thought that might have been the "smoking gun", but the same 4 errors
appeared during startup after reverting back to 12.2(55)SE1.  But
curious that it isi "4 issues" and I had exactly "4 crossed up vlans"...


Jeff