[outages] Comcast LA issue?

[Jeremy Chadwick]

Something weird is going on with Comcast in Los Angeles: small packets
appear to be getting discarded with a loss rate of ~6-10% while larger
packets (I've tried 1000 and 1200) seem to make it through without any
loss.  Things like SSH and IMAP+SSL are impacted consistently
(particularly during key exchange), while HTTP does not appear to be, so
the issue appears to be affecting both TCP and ICMP (have not tested
UDP).

First sign of this issue, according to my logs, is at 11:51 PT (UTC-8),
and has gradually gotten worse over the past few hours.

Looks like the issue is with the forward path (Source->Destination
listed below), as the return path only shows loss at the final
destination.  (I currently cannot provide a return-path mtr because the
only way I can access my box is through VNC, and I don't like doing
things like screenshots of a VNC window, so you'll just have to trust
me)

Nothing anomalous shows up when using L3's looking glass to ping things,
and the only Comcast LG I know of is in NYC.

I most likely won't see Email responses from anyone (directly or to the
list) until this problem is addressed, since my mail is hosted on shown
box and cannot currently be fetched due to the problem.  Frustrating.

Source IP: 69.181.142.213
Destination IP: 174.136.108.146

ICMP 56 bytes:

                                                                      Packets               Pings
 Host                                                               Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan (192.168.1.1)                                        0.0%   185   185   0.2   0.2   0.2   0.3
 2. 69.181.136.1                                                     0.0%   185   185   8.9   9.9   7.0  21.1
 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)   0.0%   185   185   8.7   9.9   7.9  17.0
 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134  0.0%   185   185  10.7  12.9   9.7  20.7
 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.90.93)      98.4%   184     3 49496 49006 47906 49615
 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)   5.4%   184   174  11.9  14.1  10.9  23.5
 7. 50.242.148.34                                                    6.5%   184   172  11.7  13.4  10.4  69.9
 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)             66.7%   184    61  44.9  20.6  18.0  44.9
 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)             59.0%   184    75  18.7  19.8  18.0  26.0
10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)               2.2%   184   180  20.8  63.1  18.2 293.3
11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)            6.0%   184   173  19.2  22.2  19.1  39.3
12. mambo.koitsu.org (174.136.108.146)                               6.5%   184   172  18.8  20.6  18.5  36.9

Note: Loss shown at above hops #5, #8, and #9 are normal according to
historic data (i.e. ICMP prioritisation is in effect) -- however loss at
hop #6 is new.


ICMP 1200 bytes:

                                                                      Packets               Pings
 Host                                                               Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan (192.168.1.1)                                        0.0%    43    43   0.3   0.3   0.2   0.4
 2. 69.181.136.1                                                     0.0%    43    43   9.7  10.4   8.6  14.3
 3. te-0-2-0-5-sur03.santaclara.ca.sfba.comcast.net (68.86.249.69)   0.0%    42    42  10.0  10.0   8.7  16.0
 4. te-0-0-0-13-sur03.sanrafael.ca.sfba.comcast.net (162.151.78.134  0.0%    42    42  11.7  13.3  10.3  18.9
 5. ???
 6. he-2-9-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.86.102)   0.0%    42    42  14.4  15.9  12.4  25.1
 7. 50.242.148.34                                                    0.0%    42    42  12.7  13.6  11.2  23.1
 8. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)              9.8%    42    37  20.0  20.7  18.8  25.0
 9. ae-3-80.edge6.LosAngeles1.Level3.net (4.69.144.142)              0.0%    42    42  19.1  21.1  18.9  29.9
10. ae-27-27.car2.LosAngeles1.Level3.net (4.69.201.18)               0.0%    42    42  64.3 105.7  20.9 412.5
11. GIGLINX-INC.car2.LosAngeles1.Level3.net (4.30.195.86)            0.0%    42    42  20.6  22.5  19.9  31.2
12. mambo.koitsu.org (174.136.108.146)                               0.0%    42    42  20.7  22.0  20.1  26.4

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |