[outages] Packet loss between as6453/Tata, or GBLX/nLayer

[Jeremy Chadwick]

I've been dealing with this for weeks now.  The asymmetric nature of the
problem means *4* ISPs would have to get involved to really figure out
what's causing this.  What a PITA.

* Packet loss increasing between as6453.net and Tata Communications
* Packet loss increasing between GBLX and nLayer (at the PAIX?)

Ingress traffic (work->home) shows the first problem:

Source IP: 204.8.209.251
Dest IP:   67.180.84.87

|---------------------------------------------------------------------------------- --------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|             xe1-8.core2.sv1.layer42.net -    0 |  198 |  198 |    3 |    9 |   83 |    3 |
|                  sjo-bb1-link.telia.net -    0 |  198 |  198 |    2 |    4 |   31 |    3 |
|      tata-ic-140123-sjo-bb1.c.telia.net -   20 |  109 |   88 |    3 |    8 |   21 |    8 |
|if-4-3254.tcore2.PDI-PaloAlto.as6453.net -    0 |  198 |  198 |    3 |    6 |   55 |    6 |
|   if-2-2.tcore1.PDI-PaloAlto.as6453.net -    0 |  198 |  198 |    3 |    5 |   64 |    4 |
|                           66.198.127.14 -    5 |  168 |  161 |   14 |   15 |   25 |   15 |
|pos-1-10-0-0-cr01.sanjose.ca.ibone.comcast.net -    4 |  172 |  166 |   14 |   16 |   23 |   15 |
|pos-1-8-0-0-cr01.sacramento.ca.ibone.comcast.net -    2 |  186 |  183 |   17 |   19 |   29 |   18 |
|pos-0-13-0-0-ar01.oakland.ca.sfba.comcast.net -    4 |  174 |  168 |   19 |   20 |   29 |   20 |
|te-9-8-ur04.santaclara.ca.sfba.comcast.net -    1 |  194 |  193 |   12 |   13 |  173 |   13 |
|                           68.85.191.250 -    3 |  182 |  178 |   20 |   37 |  236 |   31 |
|      c-67-180-84-87.hsd1.ca.comcast.net -    2 |  190 |  188 |   28 |   32 |   74 |   29 |
|________________________________________________|______|______|______|______|______|______|

Egress traffic (home->work) shows the 2nd problem.

Source IP: 67.180.84.87
Dest IP:   204.8.209.251
                                                  Packets               Pings
 Host                                           Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan                                  0.0%   192   192   0.2   0.5   0.2   1.0
 2. c-67-180-84-1.hsd1.ca.comcast.net            0.0%   192   192  27.1  26.2  10.3 139.9
 3. te-5-5-ur04.santaclara.ca.sfba.comcast.net   0.0%   192   192  10.3  11.9   8.0 139.2
 4. te-0-4-0-2-ar01.oakland.ca.sfba.comcast.net  0.0%   192   192  12.1  13.2  10.2  46.3
 5. pos-2-0-0-0-cr01.sacramento.ca.ibone.comcas  0.0%   192   192  12.6  15.3  12.0  86.6
 6. pos-0-4-0-0-cr01.sanjose.ca.ibone.comcast.n  0.0%   192   192  17.3  19.2  15.0  69.0
 7. pos-0-1-0-0-pe01.11greatoaks.ca.ibone.comca  0.0%   192   192  22.7  24.9  19.4 118.6
 8. 208.178.58.1                                 0.0%   192   192  23.7  25.9  17.7  88.6
 9. ae9-20G.scr3.SNV2.gblx.net                   0.0%   192   192  20.1  22.9  18.5 101.3
10. po4-20G.ar1.PAO2.gblx.net                    0.0%   192   192  51.4 116.7  18.8 1955.
11. xe-2-0-4.ar1.pao1.us.nlayer.net              5.2%   192   182  40.5  55.3  30.7 124.3
12. ae0-80g.cr1.pao1.us.nlayer.net               2.6%   192   187  38.6  54.2  28.5 128.6
13. as8121.ae0-3001.cr1.pao1.us.nlayer.net       1.0%   192   190  70.6  36.8  26.7 105.8
14. xe1-4.core1.svk.layer42.net                  2.1%   192   188  27.6  33.3  27.2  86.9
15. ???

This is not ICMP prioritisation -- I feel this latency and packet loss
when using Remote Desktop and SSH.  It's really quite annoying.

Obviously the asymmetric nature means either #1 or #2 could explain the
other.  Issue smells of a capacity problem (e.g. bottleneck), but heck
if I know. 

I look forward to insights, and/or comments from anyone at any of these
companies, including off-list if need be.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                   Mountain View, CA, US |
| Making life hard for others since 1977.               PGP 4BD6C0CB |