[outages] Issues happening at Layer42 or PAIX?

[Jeremy Chadwick]

On Fri, Jun 17, 2011 at 09:23:08PM -0700, Graham Freeman wrote:
> On 17 Jun 2011, at 20:43 GMT-0700, Jeremy Chadwick wrote:
> 
> > Hey folks, is anyone aware of any problems happening right now with
> > Layer42, or within the PAIX?
> > 
> > We're seeing issues reaching both www.google.com (going through the
> > PAIX) as well as www.facebook.com (going through the PAIX).  Packets not
> > going through the PAIX appear to be working fine.
> 
> 
> FWIW, my connectivity via Layer42's network (at their 1555 Plymouth (MV) facility) appears to be fine, including to www.'('facebook'|'google')'.com.   Same appears true via IPv4 and IPv6.
> 
> True as of 21:20 GMT-7.   I didn't get your email until just now, so perhaps there was a problem that has since cleared up.

The problem hasn't cleared up.  Instead, it appears in both cases
someone has induced reconvergence/route priority to route *around* the
PAIX.  It looks like San Jose is now exclusively in use.

Here's what mtrs look like to both Google and Facebook; compare them to
the previous.

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                 fw01.noc.sv2.tellme.com -    0 |   71 |   71 |    0 |    0 |    3 |    0 |
|         gw01.nocport.cns.sv2.tellme.com -    0 |   71 |   71 |    1 |    2 |   12 |    1 |
|      gw02.edgecross2.cns.sv2.tellme.com -    0 |   71 |   71 |    1 |    2 |   19 |    1 |
|                 gw-tellme.sv.svcolo.com -    0 |   71 |   71 |    2 |    3 |   12 |    2 |
|         xe1-5-901.core1.sv1.layer42.net -    0 |   71 |   71 |    2 |    6 |   62 |    4 |
|                 sv1.br01.sjc1.tfbnw.net -    0 |   71 |   71 |    3 |    6 |   83 |    3 |
|                 ae1.bb02.sjc1.tfbnw.net -    0 |   71 |   71 |    4 |    5 |   23 |    5 |
|                 ae2.dr01.snc5.tfbnw.net -    0 |   71 |   71 |    4 |    7 |   71 |    4 |
|             po510.csw02a.snc5.tfbnw.net -    0 |   71 |   71 |    4 |    5 |   12 |    4 |
|             www-13-02-snc5.facebook.com -    0 |   71 |   71 |    4 |    4 |   13 |    4 |
|________________________________________________|______|______|______|______|______|______|

|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                 fw01.noc.sv2.tellme.com -    0 |   21 |   21 |    0 |    0 |    2 |    1 |
|         gw01.nocport.cns.sv2.tellme.com -    0 |   21 |   21 |    1 |    1 |   12 |    4 |
|      gw02.edgecross1.cns.sv2.tellme.com -    0 |   21 |   21 |    1 |    3 |   11 |    1 |
|                 gw-tellme.sv.svcolo.com -    0 |   21 |   21 |    3 |    4 |   12 |    7 |
|         xe1-5-901.core1.sv1.layer42.net -    0 |   21 |   21 |    3 |    5 |   12 |    8 |
|           eqixsj-google-gige.google.com -    0 |   21 |   21 |    3 |    7 |   56 |   10 |
|                          216.239.49.168 -    0 |   21 |   21 |    3 |    6 |   14 |    9 |
|                          64.233.174.119 -    0 |   21 |   21 |    3 |    5 |   12 |   10 |
|                          74.125.224.144 -    0 |   21 |   21 |    3 |    4 |   12 |    8 |
|________________________________________________|______|______|______|______|______|______|

So, anyone know what's going on in the PAIX?  :-)

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |