[c-nsp] T3 or Ethernet delivery?

[Jeffrey Ollie]

On Wed, Apr 8, 2009 at 2:14 AM, Seth Mattinen <sethm at rollernet.us> wrote:
> One of my carriers has given me a choice for a new circuit delivery: T3
> or Ethernet.

I would go for Ethernet in a heartbeat.

> The cost difference between the two
> is not significant enough to be the sole deciding factor and I'm not
> using pure-Ethernet platforms so it's just a matter of adding the right
> interface card.

The cost difference on a ~40Mb/s circuit might not be much different
delivered via T3 or Ethernet, but what about anything faster?
Ethernet readily scales to 1Gb/s and 10Gb/s is not unreasonable these
days.  40Gb/s and 100Gb/s Ethernet will be here in a year or two.
Even if you start out with a 100Mb/s Ethernet port you won't have to
bond interfaces or move to more expensive electronics to go past
~40Mb/s.

> How do you detect a "down" condition on Ethernet? My experience is that
> the interface could be up/up because Ethernet doesn't know about
> anything further down the line and ends up throwing packets into a
> magical black hole. Or worse, secret packet loss.

There's nothing unique to Ethernet about that...

> Can you even troubleshoot Ethernet? Normally if I'm seeing something
> like out of frame errors or AIS, I can say "hey, there's a problem and
> it's X". It scares me to think of opening trouble tickets as "it's
> broken and I can't really tell you why".

Stick a PC directly onto your WAN connection (or stick a switch in
there and use port spanning) and run Wireshark.  Try that with a T3
connection.

> With a T3 I can be fairly certain that if there aren't any alarms that
> my end is happily talking to the other end. How does one accomplish the
> same with Ethernet?

The Ethernet protocol includes CRC checks so most hardware will detect
packet errors.  Sure, the CRC isn't perfect and you can construct
pathological examples where corrupted packets will pass the CRC checks
but

> A periodic "ping" seems rather ambiguous as a health
> check.

You'd want to do something like this anyway, since the alarms on a T3
are only a layer 1 check, pings check to make sure that things are
working at least up through layer 3.  As others have stated, running a
dynamic routing protocol across the link gives even more assurance
that packets aren't going into a black hole.

-- 
Jeff Ollie