[outages] Comcast and GTT/MZIMA LAX packet loss issue

[Jeremy Chadwick]

For the past 3 days (roughly when the issue began), I've been seeing
signs of what may be network saturation between GTT/MZIMA and Comcast in
the Los Angeles area, or alternately an overutilised router.

The issue starts approximately around 18:00 PT and continues all the way
until roughly midnight.

Here's an example of what's happening tonight/right now:

Source IP: 67.180.84.87     (Comcast; Mountain View, CA)
Dest IP:   206.125.172.42   (ARP Networks with GTT/MZIMA peering; Sylmar, CA)

 Host                                           Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan                                  0.0%   509   509   0.3   0.2   0.2   2.1
 2. c-67-180-84-1.hsd1.ca.comcast.net            0.0%   508   508  27.5  25.8   9.1 587.6
 3. te-0-0-0-12-ur05.santaclara.ca.sfba.comcast  0.0%   508   508  10.0  10.9   8.1  46.6
 4. te-1-1-0-0-ar01.sfsutro.ca.sfba.comcast.net  0.0%   508   508  17.6  18.2   9.9  49.8
 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.ne  0.0%   508   508  17.6  18.6  11.3  36.8
 6. pos-2-9-0-0-cr01.losangeles.ca.ibone.comcas  0.0%   508   508  21.1  23.3  18.9  57.9
 7. 75.149.231.238                               4.1%   508   487  22.7  29.3  21.1  56.9
 8. ge0-arpnet.cust.lax07.mzima.net              0.0%   508   508  22.9  29.0  20.1 234.9
 9. omake.koitsu.org                             3.3%   508   491  25.9  25.1  21.2  56.8


Source IP: 206.125.172.42   (ARP Networks with GTT/MZIMA peering; Sylmar, CA)
Dest IP:   67.180.84.87     (Comcast; Mountain View, CA)

 Host                                           Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. 206.125.172.41                               0.0%   534   534   0.9   6.7   0.8 197.0
 2. ge0-15.as01.lax07.mzima.net                  0.0%   533   533  13.2   5.2   0.5  19.1
 3. be-10-201-cr01.losangeles.ca.ibone.comcast.  3.9%   533   512   8.1   8.1   3.9  65.8
 4. pos-2-9-0-0-cr01.sanjose.ca.ibone.comcast.n  3.4%   533   515  15.6  13.7  10.8  25.8
 5. he-0-6-0-0-ar01.sfsutro.ca.sfba.comcast.net  5.3%   533   505  17.5  18.7  12.2  25.4
 6. te-0-6-0-0-ur05.santaclara.ca.sfba.comcast.  5.3%   533   505  14.8  14.4  13.1  15.7
 7. te-18-10-cdn31.santaclara.ca.sfba.comcast.n  4.5%   533   509  27.9  28.6  15.1 127.9
 8. c-67-180-84-87.hsd1.ca.comcast.net           6.2%   533   500  25.2  24.1  20.9  50.3

I keep 60 days worth of periodic mtrs, run for 40 seconds at 60 second
intervals, so if more historic data is needed, let me know.

Verification and/or any insights would be appreciated.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |