[c-nsp] Ethernet debugging...MTU?

[Andy Dills]

Ok, I have a weird situation that I'd like to get some input on.

Cliff notes:

Is the inability to get ping replies with datagrams of larger than 9216 
bytes across a 100mbps ethernet circuit an indication that the far end is 
setup with an MTU consistent with jumbo frames? 

What to do if the far end swears it's set for 1500?


Background:

We're in the process of turning up a new fast-e to a company located in 
Equinix. We are not located in Equinix, but Level3 is, and we have Level3 
fiber in our datacenter. There are two major segments of this circuit: the 
long haul to Ashburn and the cross connect in Equinix. The run at Equinix 
is too long for copper, and Level3 for some reason insisted upon a copper 
handoff, so Equinix supplied and installed transceivers to enable a fiber 
x-con that is delivered via copper to the cage.

In our datacenter, the ethernet circuit is connected directly via a short 
copper run from Level3's space to a standard FE port on a Cisco router, 
that has previously been in use and is known to be working. If it matters, 
for now it's on a PA-2FEISL-TX (but will later be moved to a more current 
PA when put into production...to my knowledge, even though that PA isn't 
ideal, it should still work fine at lower bandwidth levels and with proper 
full-duplex, etc. The previously attached customer had no problems pushing 
>30mbps, for example).

When testing the circuit, using the cisco ping utility, with datagrams of 
9216 bytes or less, we have no packet loss. When datagrams larger than 
9216 bytes are used, we have 100% packet loss.

Given the packet size when total failure occurs, my first reaction is that 
the other company has somehow misconfigured their switch to use jumbo 
frames, as if the circuit was a gig-e.

According to the company at the far end, their device doesn't even support 
jumbo frames, and other people are attached and working fine. They seem to 
be quite certain the problem isn't on their end. I have a hard time not 
believing them; checking the MTU is a pretty cut and dry thing, and I'm 
working with senior level people at the other company, who I'm assuming 
(perhaps an incorrect assumption) run and manage their international 
network.


To narrow down the problem, we have had Level3 setup a laptop in their 
cage at Equinix, attached to the interface facing our datacenter. When 
testing to that, I had no packet loss with packets of any size up to the 
maximum of 18024 bytes. To me this eliminates the long-haul portion from 
consideration.

We've also had Equinix double check (at the supervisor level) that the 
transceivers are 10/100, hardcoded for 100 full-duplex (as is everything 
else end to end). I also have a hard time believing that Equinix would 
have any difficulties installing the correct model and properly configured 
ethernet transceivers; Ashburn is a top notch facility with good people. 
(I was hoping with my fingers crossed they had accidently installed gig-e 
transceivers, or that Level3 had accidentally ordered gig-e 
transceivers...no luck).

As this has been a lingering issue for some time, I'm currently pushing 
for technical reps from all of the companies involved to meet up at 
Equinix, sit in a room, and figure it out. This is a bit difficult for the 
other company as they don't have any real local staffing. So, I'm hoping 
to come up with a solution that doesn't involve them getting on an 
airplane, but it's starting to look like our only avenue of resolution.

That said, has anybody encountered this before, or has any theories about 
ways I can debug this short of having the other company (who doesn't have 
local staff) visit their cage and attach a laptop facing us, to localize 
the issue to either their switch or the Equinix cross connect?

Am I likely correct in my theory that something is configured for the 
jumbo frame MTU and thus response packets aren't being properly 
fragmented?

Thanks for any insight.

Andy

---
Andy Dills
Xecunet, Inc.
www.xecu.net
301-682-9972
---