[nsp] Gig E problem

From: Jim Warner (warner@cats.UCSC.EDU)
Date: Sat Apr 28 2001 - 16:34:59 EDT


We are getting ratty performance on a new Gig-E point-to-point link
between our 7507 and Catalyst 5500.

Prior to bringing up the Gig-E link, the connection between these devices
was fast ethernet. Now, with OSPF set to prefer GE, it looks like:

              7507 Cat5500

         ------------ ---------------
         | VIP4-80, | Fast-E |WS-X5213A |
         | slot 0 |-----------------|port 5/2 |
         | | | |
         | Gig-E | LX Gig-E |WS-X5403 |
         | slot 1 |-----------------|port 3/1 |
         | | | |
         | | |Route-Switch |
         | RSP 4 | |Mod slot 4 |
         | | | |
         ------------ ---------------

We have rigged some static routes so we can perform four different
ping tests: (out Gig in FE, out Gig in Gig, out FE in FE, out FE in Gig)

Ping tests that take the Gig-E path from the 5500 --> 7500 are
slow and have a high round trip variance. Response times on
the other paths are sub millisecond. From IOS perspective:

Gig-E
Success rate is 100 percent (3341/3341), round-trip min/avg/max = 1/25/260 ms

Fast-E
Success rate is 100 percent (1000/1000), round-trip min/avg/max = 1/1/12 ms

The behavior holds with delay probes through the routers (to/from unix
computers) where the probe packets are not process switched. In all
cases, we are not seeing packet loss, only delay. The amount of delay
appears to correlate with the average traffic through the routers -- 25
mS of "extra" delay corresponds to about 10 Mb/s and 2500 packets/s.

To see the effect of switching from FE to GE this morning, see:

  http://noc.ucsc.edu/rt-delay.gif

We are puzzled. None of the error counters we normally look at are
incrementing. Flow control is off everywhere. Duplex is full everywhere.
I guess I'm looking for suggestions of really-dumb-things-I-forgot-to-check.

Thanks in advance... -jim warner, UCSC



This archive was generated by hypermail 2b29 : Sun Aug 04 2002 - 04:12:35 EDT