[outages] Chronic packet loss/latency in LAX (4 providers potentially involved)

[Jeremy Chadwick]

Hi folks,

For the past 3-4 weeks (likely longer), there has been a peering point
in Los Angeles that has been repeatedly overtaxed every night for
roughly 5-6 hours.  Specifically high latency and packet loss.

It tends to start at roughly the same time every day (1800 PDT), and
tends to dissipate around the same time as well (2300 PDT or so) -- i.e.
prime-time.  This is not ICMP prioritisation -- this is real packet loss
(I can tell via SSH, fetching mail (IMAP), etc.).  However in recent
days (past week or so), the issue has persisted even until hours as late
as 0100 PDT, and today (for example) has existed since as early as 1500
PDT.  So the situation as I said is getting worse.

Because of asymmetrical routing, it's impossible for me to tell where
peering-wise the actual issue is:

a) between Comcast and GTT/nLayer
b) between Comcast and Tata/AS6453
c) between NTT/Verio and Tata/AS6453

Of those 4 providers I've only a relationship with one; me getting all
four to talk simultaneously, while the issue is happening, is virtually
impossible.  However it happens every night like clockwork, for a long
period of time, so resolution shouldn't be all that difficult.

I do periodic mtrs (think traceroute + ping combined) between my home
connection and my VPS.  Probe durations are 40 seconds, at intervals of
60 seconds.  I store roughly 2-3 months of data, and I can make all this
data available.

Here's examples taken from tonight, both directions.  Source and
destination IPs are provided.


Source IP: 67.180.84.87    (Comcast; Mountain View, CA)
Dest IP:   206.125.172.42  (ARP Networks; Sylmar, CA)

=== Mon May 27 19:36:00 PDT 2013  (1369708560)
HOST: icarus.home.lan             Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- 192.168.1.1                0.0%    40    40   0.3   0.3   0.2   0.4
  2.|-- 67.180.84.1                0.0%    40    40  11.4  24.9  10.1  61.7
  3.|-- 68.85.191.249              0.0%    40    40  11.3  11.7   8.9  27.7
  4.|-- 69.139.199.106             0.0%    40    40  14.6  13.3  10.4  16.1
  5.|-- 68.86.91.45                0.0%    40    40  15.9  19.6  13.1  34.3
  6.|-- 68.86.88.58                0.0%    40    40  25.0  22.8  20.0  27.2
  7.|-- 68.86.88.190               0.0%    40    40  21.5  21.9  19.6  38.2
  8.|-- 173.167.57.138            10.0%    40    36  68.5  79.4  54.1  92.3
  9.|-- 69.31.127.141              0.0%    40    40  81.4  95.1  64.5 128.8
 10.|-- 69.31.127.130             35.0%    40    26  69.5  69.6  43.0  96.1
 11.|-- 69.174.121.73              0.0%    40    40  67.0  72.3  39.6  90.4
 12.|-- 67.199.135.102            10.0%    40    36  39.1  49.2  36.7 249.8
 13.|-- 206.125.172.42             5.0%    40    38  37.1  36.0  34.2  49.9
=== END

DNS resolution for relevant hops:

$ host 68.86.88.190
190.88.86.68.in-addr.arpa domain name pointer pos-0-4-0-0-pe01.600wseventh.ca.ibone.comcast.net.

$ host 173.167.57.138
Host 138.57.167.173.in-addr.arpa. not found: 3(NXDOMAIN)

$ host 69.31.127.141
141.127.31.69.in-addr.arpa domain name pointer ae0-110g.cr1.lax1.us.nlayer.net.



Source IP: 206.125.172.42  (ARP Networks; Sylmar, CA)
Dest IP:   67.180.84.87    (Comcast; Mountain View, CA)

=== Mon May 27 19:37:00 PDT 2013  (1369708620)
HOST: omake.koitsu.org            Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- 206.125.172.41             0.0%    40    40   1.5  12.8   0.9 181.8
  2.|-- 208.79.88.135              0.0%    40    40   0.6   1.0   0.5  11.6
  3.|-- 129.250.198.185            0.0%    40    40   0.8   0.9   0.7   1.4
  4.|-- 129.250.2.221              0.0%    40    40   1.0   1.0   0.9   1.5
  5.|-- 64.86.252.65              32.5%    40    27   0.5   0.8   0.5   8.6
    |  `|-- 216.6.84.65
  6.|-- 173.167.59.185            55.0%    40    18  19.6  19.3  15.7  75.1
  7.|-- 68.86.82.61               40.0%    40    24  17.9  17.5  15.9  19.5
  8.|-- 68.86.88.57                5.0%    40    38  25.2  25.7  24.1  27.5
  9.|-- 68.86.90.94                7.5%    40    37  25.8  26.6  24.7  28.7
 10.|-- 69.139.198.81              5.0%    40    38  26.6  26.6  26.3  27.0
 11.|-- 68.85.191.254              7.5%    40    37  47.9  46.0  30.5  50.5
 12.|-- 67.180.84.87               7.5%    40    37  36.0  36.1  34.0  50.3
=== END

DNS resolution for relevant hops (hop #5 indicates round-robin'ing
between two different interfaces presumably on the same Tata/AS6453
router):

$ host 129.250.2.221
221.2.250.129.in-addr.arpa domain name pointer ae-3.r05.lsanca03.us.bb.gin.ntt.net.

$ host 64.86.252.65
65.252.86.64.in-addr.arpa domain name pointer ix-11-2-1-0.tcore2.LVW-LosAngeles.as6453.net.

$ host 216.6.84.65
65.84.6.216.in-addr.arpa domain name pointer ix-9-1-2-0.tcore2.LVW-LosAngeles.as6453.net.

$ host 173.167.59.185
185.59.167.173.in-addr.arpa domain name pointer xe-1-2-0-0-pe01.onewilshire.ca.ibone.comcast.net.


If relevant providers are on this list and can respond, that would
be awesome.  If they can't respond but can instead start the ball
rolling on rectifying this, I think myself and many other 'netizens
would really appreciate it.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |