[c-nsp] 3750G-48 / 12.2(58)SE1 issue (word of warning)

[Tóth András]

Hi Jeff,

This seems to be caused by CSCsu83001 and CSCsv11710. This is fixed in
12.2(50)SE, however if you upgrade the switch from older versions, you
will see this problem. I understand you did an upgrade to 12.2(58)SE1,
but if you had older versions running on the switches, you will face
this issue.

The solution is to remove the "snmp ifmib ifindex persist"
configuration from the switch prior to the upgrade, because the old
version uses a different structure and after the upgrade, it will
still use the original structure. Also, make sure you don't have
"snmp-server ifindex persist" in your configuration, because it's not
supported. The switch will retain physical interfaces ifindex after
upgrade even without this configuration item as physical interfaces
use pre-defined index based on the switch number and type of
interface.

Best regards,
Andras


On Sat, May 21, 2011 at 8:42 PM, Jeff Kell <jeff-kell at utc.edu> wrote:
> We had a "need" (bugfix) to upgrade our 12.2(55)SE1 stack of 4 *
> 3750G-48s to 12.2(58)SE1 (IP Services).  The maintenance window was last
> night, and appeared to go well, until the phone lit up...
>
> To make a very long story as short as possible, to the best extent I can
> figure it out...
>
> The stack is running several VRFs, incoming traffic from the core
> arriving on a 4-port cross-stack etherchannel trunk carrying multiple
> vlans (one per VRF), with connectivity to the edge (default) out of the
> core to our border routers.  All traffic (in-house as well as default)
> traverses the same trunks.  IGP on the VRFs is EIGRP in all cases.
>
> The borders lit up after the upgrade hitting reverse-path verify on our
> ASAs.  Two of the VRFs were showing traffic from "other" VRFs where they
> should never be seen.
>
> It appears that ingress traffic on one vlan was landing on another
> (and/or else egress traffic on one vlan was landing on another).  The
> "crossed-plumbing" was exactly two pairs out of a possible 12, with
> improper egress resulting on 2 destination vlans.  Everything else was fine.
>
> If I shutdown the SVIs of the two problem vlans, everything worked fine
> again (the "other two" vlans of the pair then routed correctly).
>
> I checked default routes, CEF adjacencies, everything I could think
> could be affecting things, but if either of the problem SVIs was brought
> back up, the mess ensued again.
>
> Rebooting the stack resulted in no change (well, actually, only one pair
> was crossed initially, the second pair appeared after the second reboot).
>
> Reloaded 12.2(55)SE1, brought the two SVIs back up, all was fine.
>
> Still scratching my head.  I also applied 12.2(58) to a 3750E and 3750X
> stack during the same window, no issues, very similar configuration and
> adjacencies.
>
> So for the 3750G platform, you might hold off, or at least cautiously
> approach this update.  I have a TAC case on this one but it may be a
> tough one to reproduce (I certainly don't want to repeat it here!).
>
> The only "anomaly" during the update was the appearance of the following
> errors during the reload (same 4 appeared in each of the 4 member's
> startup logs):
>
>> *Mar  1 00:04:14.938: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been
>> ADDED to the stack
>> *Mar  1 00:04:14.938: %STACKMGR-4-SWITCH_ADDED: Switch 2 has been
>> ADDED to the stack
>> *Mar  1 00:04:14.938: %STACKMGR-4-SWITCH_ADDED: Switch 3 has been
>> ADDED to the stack
>> *Mar  1 00:04:14.938: %STACKMGR-4-SWITCH_ADDED: Switch 4 has been
>> ADDED to the stack
>> *Mar  1 00:04:15.248: platform assert failure: hwidb->snmp_if_index ==
>> ifIndex: ../src-hulc/src-common/hpm_idbman.c: 1843:
>> hpm_register_idb_with_snmp
>> *Mar  1 00:04:15.248: -Traceback= 229B110z 16AEC38z 184DBA8z 184F040z
>> 184E298z 184D340z 1864174z 1832F7Cz 16B6824z 3112CECz 3112D90z
>> 3112EF8z 22A6F50z 22A721Cz 1CC0FA8z 1CBAE1Cz
>> *Mar  1 00:04:15.340: platform assert failure: hwidb->snmp_if_index ==
>> ifIndex: ../src-hulc/src-common/hpm_idbman.c: 1843:
>> hpm_register_idb_with_snmp
>> *Mar  1 00:04:15.340: -Traceback= 229B110z 16AEC38z 184DBA8z 184F040z
>> 184E298z 184D340z 1864174z 1832F7Cz 16B6824z 3112CECz 3112D90z
>> 3112EF8z 22A6F50z 22A721Cz 1CC0FA8z 1CBAE1Cz
>> *Mar  1 00:04:15.407: platform assert failure: hwidb->snmp_if_index ==
>> ifIndex: ../src-hulc/src-common/hpm_idbman.c: 1843:
>> hpm_register_idb_with_snmp
>> *Mar  1 00:04:15.416: -Traceback= 229B110z 16AEC38z 184DBA8z 184F040z
>> 184E298z 184D340z 1864174z 1832F7Cz 16B6824z 3112CECz 3112D90z
>> 3112EF8z 22A6F50z 22A721Cz 1CC0FA8z 1CBAE1Cz
>> *Mar  1 00:04:15.491: platform assert failure: hwidb->snmp_if_index ==
>> ifIndex: ../src-hulc/src-common/hpm_idbman.c: 1843:
>> hpm_register_idb_with_snmp
>> *Mar  1 00:04:15.491: -Traceback= 229B110z 16AEC38z 184DBA8z 184F040z
>> 184E298z 184D340z 1864174z 1832F7Cz 16B6824z 3112CECz 3112D90z
>> 3112EF8z 22A6F50z 22A721Cz 1CC0FA8z 1CBAE1Cz
>
> I thought that might have been the "smoking gun", but the same 4 errors
> appeared during startup after reverting back to 12.2(55)SE1.  But
> curious that it isi "4 issues" and I had exactly "4 crossed up vlans"...
>
>
> Jeff
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>