[c-nsp] Unusual Problem with Catalyst 6500 and Sup702-3b

[Stephan M. Mackenzie]

qos3outlost is 0's across the board

# show int  | i drops
Input queue: 0/75/90/90 (size/max/drops/flushes); Total output drops: 87

There was only 1 interface with output drops...

And no interfaces with errors more than 1 ( frame errors )

I have a spare chassis, and I am expecting the 6748-SFP card tomorrow, so I
might try to bring up a test environment and see if I can replicate the
issue or see if it goes away...

do you think there is any problem with the MSFC3 cards being different
versions?

2.3 vs 2.5

If the ports had mismatch, would it be possible for 1 test from say Toronto
to run slow, and a test on the same server from Amsterdam to run fast?

I have a networking contractor coming tomorrow to look at it, I have a
feeling though its going to take some process of elimination to identify the
buggy piece of the puzzle.

I wish I had a spare Sup720, I may just pull out the standby card to test in
the other chassis.

Or pick up a card, the market seems good for 3BXLs right now

Thanks again for your ideas and help

Stephan


-----Original Message-----
From: Nick Hilliard [mailto:nick at foobar.org] 
Sent: Sunday, October 17, 2010 4:47 PM
To: Stephan M. Mackenzie
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Unusual Problem with Catalyst 6500 and Sup702-3b

On 17/10/2010 23:11, Stephan M. Mackenzie wrote:
> I spot checked a few ports

why not check them all?  Electrons are cheap.

% perl -e 'for($i=1;$i<=2;$i++){for($j=1; $j<=48;$j++){print"show counters
interface gigabitEthernet $i/$j | i qos3Outlost\n"}}'

Cut-n-paste the output of this into your switch.[1]

Also, have you checked all of the interfaces for port drops?

# show int  | i drops

If you see a port with lots of drops, check to make sure that the port
drops aren't increasing.

> the one that had a few was my main link to
> Provider...
> 
> edge01.stl#show counters interface gigabitEthernet 6/2 | i qos3Outlost
> 53.                        qos3Outlost = 87

that's pretty insignificant.

> if I run MTR or extended pings to the slower hosts, I dont see any
> measureable packet loss

Probably not, no.  mtr doesn't fill a switch port up with traffic.  Nor
does ping, unless you're running a flood ping.

> is there any way to debug a network request

What is a "network request"? And how does it relate to individual packets?
 This is a switch router.  It doesn't care about layer 4 through layer 9.

If you want to snoop on traffic, check out span / rspan / erspan.  Beware
that on a loaded 6148 blade, this can cause havoc with dropped packets.

If both your backplane drops and your port queue drops are insignificant,
then check for duplex mismatches (as other people have suggested), framing
errors and that sort of thing.  e.g.

# show int  | i error

Under normal circumstances, you should see 0 errors, low and non-increasing
drops and 0 backplane drops.

Nick

[1] we're all lazy.  here's the output: http://pastebin.com/PqAEZgNN