[c-nsp] Unusual Problem with Catalyst 6500 and Sup702-3b

[Stephan M. Mackenzie]

I spot checked a few ports, the one that had a few was my main link to
Provider...

edge01.stl#show counters interface gigabitEthernet 6/2 | i qos3Outlost
53.                        qos3Outlost = 87

I now know we wasted our money upgrading the memory on the 3B cards...

if I run MTR or extended pings to the slower hosts, I dont see any
measureable packet loss

is there any way to debug a network request

Like log the whole transaction based on source and destination IP

Appreciate your help and advice

Stephan

-----Original Message-----
From: Nick Hilliard [mailto:nick at foobar.org] 
Sent: Sunday, October 17, 2010 2:36 PM
To: Stephan M. Mackenzie
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Unusual Problem with Catalyst 6500 and Sup702-3b

On 17/10/2010 19:32, Stephan M. Mackenzie wrote:
> For example,  testing from Toronto, the fast servers run at about
> 5.5M/s, the slow servers start at 500K/s -  1.1MB's then settle down to
> as low as half that speed.

That looks like packet loss.

> At peak load the router is pushing 550mb/s, cpu is at 2% and memory at
> 11% ( memory has been upgraded to 1gb/1gb )

incidentally, upgrading the memory on a sup720 is almost certainly a
complete waste of time and money unless you also upgrade the tcam (from 3b
to 3bxl) - in which case you get the extra memory as part of the upgrade
package.

As you're using ws-x6148 cards and you're experiencing packet loss, you
need to be aware that these line cards are desktop aggregation line cards
and are _not_ intended for server farms.  Please see the following URL:

>
http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note0918
6a00801751d7.shtml#ASIC

This means that internally, this line card acts a little like 6 gigabit
ethernet hubs, each connected to the c6500 backplane with a 1Gb uplink.  It
is very easy to run into packet loss problems on this configuration.

You didn't indicate in your email which ports these servers were connected
up to, but assuming that they are connected into the ws-x6148, you can find
out if your box is experiencing backplane drops by executing the following
command for each port:

#show counters interface gigabitEthernet x/y | i qos3Outlost

If this shows that you're seeing packet loss, then either you need to
rearrange the port configuration on your box so that none of the port
groups (1-8, 9-16, 17-24, 25-32, 33-40, 41-48) bursts above 1G.
Alternatively, use a ws-x6748 line card instead.

Terrible performance when using ws-x6148 line cards is a FAQ.  Please see
the mailing list archives for more information.

Nick