[c-nsp] Weird problem on Cat6509 + WS-X6348-RJ-45

[Brad Swanson]

Jeff,
6348 = trouble for us.  I just spent Friday morning (4 hours) and Saturday 
morning (4 hours) figuring out that the line card, a 6348, was tieing up the 
backplane on a 6509 and that the code we are running (6.4(4a)) does not 
detect this scenario and so the switch ends up resetting itself.  Various 
hidden cmds do show if the backplane is being tied up, TAC of course has the 
cmds...to be used wisely.  Newer code auto-detects this issue and shuts down 
the card via power drop.

Also this past week our server group complained of throughput on a 6348, yet 
a sho test 5 and sho port stat showed the card clean.  I did a soft reset 
since the servers were set up redundantly, and of course the card came back 
faulty...pinnacle error.  I have seen this type of failure in what you 
describe also, the card just stops working without any syslog messages or 
other info being relayed.

These are just two examples of what seems to be a 6348-whoops in design, at 
least to me it is.

My advice, replace the card...

Thanks,
Brad


>From: Tim Stevenson <tstevens at cisco.com>
>To: Jeff Kell <jeff-kell at utc.edu>, cisco-nsp <cisco-nsp at puck.nether.net>
>Subject: Re: [c-nsp] Weird problem on Cat6509 + WS-X6348-RJ-45
>Date: Sun, 25 Sep 2005 21:24:31 -0700
>
>Well, from the 6500 side, there is a lot more to look at than just
>the ARP cache.
>
>There are many platform specific commands that should be issued from
>the 6500 side when the problem is happening. TAC should know what
>these commands are (if they don't, then escalate the case, you're
>being outsourced).
>
>You haven't mentioned the sup version - at a minimum you want to look
>at either the MLS cache or the hardware FIB & ADJ entries, maybe the
>L2 table, etc.
>
>Tim
>
>At 08:12 PM 9/25/2005, Jeff Kell mused:
> >I have now been through the *third* occurrance of a very strange
> >communications hangup involving a WS-X6348 blade  and, at various
> >stages, a PIX 515E and a 2611.  It is one of those non-reproduceable
> >things that is hard to troubleshoot (at this point TAC would
> >probably shrug this one off).  Any clues appreciated.  I'll try to
> >make a long story short...
> >
> >Communication stops on a point-to-point (/30) link with the
> >neighboring device (100TX link on both ends of a short jumper
> >cable).  No ping response from either end.  Tried changing
> >cables.  Tried changing ports.  Tried clearing ARP.  Tried clearing
> >mac-address tables.  Tried clearing interfaces.  No dice.  The other
> >device can not communicate with the 6509.  If relevant, IOS is 
>12.1(26)E1.
> >
> >Other interfaces, routing protocols, operation are unaffected.
> >
> >6500 sees the neighbor and has it in ARP table.
> >Other end does not see the 6509's MAC (ARP entry 'incomplete').
> >Tried static MAC in the other end to no avail.
> >
> >Reloading the 6509 'often' clears it, power-cycling always does.
> >Reloading the other device almost never clears it.
> >Power-cycling the other device sometimes clears it.
> >
> >I can't reliably reproduce it, and since this is production
> >hardware, I can't spend much time troubleshooting it (unless the
> >'other' device is not critical, as was the case with the 2611 recently).
> >
> >Ring any bells?
> >
> >Jeff
> >_______________________________________________
> >cisco-nsp mailing list  cisco-nsp at puck.nether.net
> >https://puck.nether.net/mailman/listinfo/cisco-nsp
> >archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
>
>Tim Stevenson, tstevens at cisco.com
>Routing & Switching CCIE #5561
>Technical Marketing Engineer, Catalyst 6500
>Cisco Systems, http://www.cisco.com
>IP Phone: 408-526-6759
>********************************************************
>The contents of this message may be *Cisco Confidential*
>and are intended for the specified recipients only.
>_______________________________________________
>cisco-nsp mailing list  cisco-nsp at puck.nether.net
>https://puck.nether.net/mailman/listinfo/cisco-nsp
>archive at http://puck.nether.net/pipermail/cisco-nsp/