[outages] gblx issues Europe to US

[Jeremy Chadwick]

On Thu, Oct 25, 2012 at 07:32:31PM +0200, Gert Doering wrote:
> Hi,
> 
> On Thu, Oct 25, 2012 at 03:17:26PM +0000, Erick Caldera wrote:
> > Tracing the route to IP node (206.165.73.78) from 1 to 30 hops
> > 
> >   1    <1 ms   <1 ms   <1 ms TenGigabitEthernet7-3.ar2.MAD1.gblx.net [64.215.82.125]
> >   2    *       *       *     ?
> >   3    *       *       *     ?
> >   4    *       *       *     ?
> >   5    *       *       *     ?
> 
> Yeah, this is exactly why we terminated our contract with GBLX.
> 
> Frequent black holes in their MPLS network, and no traceroute capability to
> be able to pinpoint the point of brokenness.
> 
> (Worse, it might even be the customer egress router on the other side
> that's missing a return route(!), but you can't see whether it does inside
> GBLX or not)

Some clarification: I've seen two types of MPLS in use: layer 2 and
layer 3.

Layer 2 MPLS (ex. Abovenet "long-haul" setups) behaves like what's
described above; you have no visibility into the MPLS network via a
traceroute -- even if done on the device where the circuit terminates;
thus naturally traceroute isn't going to see anything "in between".

Layer 3 MPLS (ex. AT&T "long-haul" setups) provides visibility as long
as the traceroute is done from the device where the circuit (Ethernet,
SONET, etc.) terminates.  In this case, traceroute from the terminating
device will provide visibility within the MPLS network.  An added bonus
is when the traceroute utility supports MPLS tag decoding.  :-)

I had to deal with this ordeal on a near-weekly basis at a past job
where both types of MPLS were in use simultaneously (separate circuits
of course).  L2 MPLS requires -- no, *demands* -- the carrier be on the
ball about monitoring their circuits and being able to very quickly,
while on the phone, trace everything back to a common point.

The only advice I can give regarding L2 MPLS is after every outage,
have a long phone conversation with the carrier and set expectations.
Do this every single time.  Get network engineers and your account rep
on the call.  I speak from experience when I say this is the only way
to get L2 MPLS providers to improve.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |