[outages] Comcast issues at Great Oaks?

Jeremy Chadwick jdc at koitsu.org
Thu Jun 9 01:23:23 EDT 2016


Issue appears to have started as early as 20:00 PDT but I need to review
more logs to see if occasional trickles of problems began even earlier.
mtrs from both direction are provided and src/dst IPs.  These are ICMP
echo/echo-reply with a 512-byte ICMP payload (vs.  default 56).

Issue continues as I write this.


src IP: 69.181.142.213
dst IP: 104.238.183.73

=== Wed Jun  8 22:04:00 PDT 2016  (1465448640)
Start: Wed Jun  8 22:04:00 2016
HOST: icarus.home.lan                                                  Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- gw.home.lan (192.168.1.1)                                         0.0%    30    30   0.3   0.3   0.2   0.4
  2.|-- 96.120.89.145                                                     0.0%    30    30   9.2   9.6   8.7  12.4
  3.|-- be-20003-sur04.santaclara.ca.sfba.comcast.net (68.86.249.249)     0.0%    30    30   9.7   9.8   8.9  12.6
  4.|-- hu-0-3-0-6-ar01.hayward.ca.sfba.comcast.net (68.87.192.249)       3.3%    30    29  10.8  10.9  10.3  15.8
  5.|-- hu-0-2-0-1-ar01.santaclara.ca.sfba.comcast.net (68.85.57.145)     3.3%    30    29  11.2  11.1  10.4  13.5
  6.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93)        3.3%    30    29  11.1  11.2  10.3  15.3
  7.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158)      0.0%    30    30  14.6  15.7  12.9  36.3
  8.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 33.3%    30    20  13.2  13.5  12.1  21.9
  9.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169)                          36.7%    30    19  12.7  15.5  11.9  64.7
 10.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246)                      36.7%    30    19  12.5  17.7  11.8  87.9
 11.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90)       40.0%    30    18  19.8  16.4  12.3  66.1
 12.|-- ???                                                              100.0    30     0   0.0   0.0   0.0   0.0
 13.|-- mambo.koitsu.org (104.238.183.73)                                41.4%    29    17  13.2  13.3  12.5  19.9
=== END

=== Wed Jun  8 22:05:00 PDT 2016  (1465448700)
Start: Wed Jun  8 22:05:00 2016
HOST: icarus.home.lan                                                  Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- gw.home.lan (192.168.1.1)                                         0.0%    30    30   0.4   0.3   0.3   0.4
  2.|-- 96.120.89.145                                                     0.0%    30    30   9.0   9.5   8.5  12.5
  3.|-- be-20003-sur04.santaclara.ca.sfba.comcast.net (68.86.249.249)     0.0%    30    30   9.3   9.7   9.0  12.3
  4.|-- hu-0-3-0-6-ar01.hayward.ca.sfba.comcast.net (68.87.192.249)       0.0%    30    30  16.9  11.6  10.3  20.8
  5.|-- hu-0-2-0-1-ar01.santaclara.ca.sfba.comcast.net (68.85.57.145)     0.0%    30    30  10.6  11.3  10.3  14.1
  6.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93)       33.3%    30    20  18.3  12.6  10.2  38.7
  7.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158)      0.0%    30    30  13.7  14.2  12.6  31.1
  8.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 40.0%    30    18  12.6  13.4  11.9  24.4
  9.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169)                          50.0%    30    15  12.5  12.6  11.8  15.3
 10.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246)                      50.0%    30    15  12.7  12.5  12.1  13.4
 11.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90)       43.3%    30    17  12.4  13.3  12.1  17.6
 12.|-- ???                                                              100.0    30     0   0.0   0.0   0.0   0.0
 13.|-- mambo.koitsu.org (104.238.183.73)                                46.7%    30    16  12.8  14.5  12.4  20.4
=== END

=== Wed Jun  8 22:06:00 PDT 2016  (1465448760)
Start: Wed Jun  8 22:06:00 2016
HOST: icarus.home.lan                                                  Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- gw.home.lan (192.168.1.1)                                         0.0%    30    30   0.4   0.3   0.2   0.4
  2.|-- 96.120.89.145                                                     0.0%    30    30   9.0   9.5   8.6  15.6
  3.|-- be-20003-sur04.santaclara.ca.sfba.comcast.net (68.86.249.249)     0.0%    30    30   9.5  10.8   9.1  19.9
  4.|-- hu-0-3-0-6-ar01.hayward.ca.sfba.comcast.net (68.87.192.249)       3.3%    30    29  10.9  11.0  10.2  14.2
  5.|-- hu-0-2-0-1-ar01.santaclara.ca.sfba.comcast.net (68.85.57.145)     3.3%    30    29  11.7  12.2  10.3  27.8
  6.|-- be-33651-cr01.sunnyvale.ca.ibone.comcast.net (68.86.90.93)       93.3%    30     2  12.9  12.2  11.5  12.9
  7.|-- be-10925-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158)      0.0%    30    30  12.5  14.2  12.1  18.1
  8.|-- hu-0-11-0-0-pe03.11greatoaks.ca.ibone.comcast.net (68.86.85.238) 36.7%    30    19  12.2  15.0  12.1  50.8
  9.|-- ae12.sjc12.ip4.gtt.net (173.205.58.169)                          33.3%    30    20  12.1  12.8  11.8  19.4
 10.|-- xe-1-1-2.sjc20.ip4.gtt.net (89.149.184.246)                      43.3%    30    17  12.1  12.5  11.9  14.9
 11.|-- as20473.xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.90)       40.0%    30    18  12.3  19.7  12.2  74.3
 12.|-- ???                                                              100.0    30     0   0.0   0.0   0.0   0.0
 13.|-- mambo.koitsu.org (104.238.183.73)                                40.0%    30    18  12.8  12.7  12.1  13.4
=== END

(Note: in the above, hop #6 packet loss is totally normal)



src IP: 104.238.183.73
dst IP: 69.181.142.213

=== Wed Jun  8 22:04:00 PDT 2016  (1465448640)
Start: Wed Jun  8 22:04:01 2016
HOST: mambo.koitsu.org                                                Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- ???                                                             100.0    30     0   0.0   0.0   0.0   0.0
  2.|-- 45.63.80.1                                                       0.0%    30    30   0.4   1.6   0.3  32.3
  3.|-- xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.89)               0.0%    30    30   0.9   2.6   0.3  15.3
  4.|-- xe-7-0-7.sjc12.ip4.gtt.net (89.149.184.237)                      0.0%    30    30   0.7   0.8   0.6   2.6
  5.|-- as7922.sjc12.ip4.gtt.net (173.205.58.170)                        0.0%    30    30   1.2   1.4   1.0   6.4
  6.|-- hu-0-4-0-5-cr01.9greatoaks.ca.ibone.comcast.net (68.86.88.205)  36.7%    30    19   2.7   2.6   1.3   4.8
  7.|-- be-7922-ar01.hayward.ca.sfba.comcast.net (68.86.94.154)         30.0%    30    21   2.6   2.9   2.5   5.9
  8.|-- hu-0-4-0-1-sur04.santaclara.ca.sfba.comcast.net (68.87.192.254) 16.7%    30    25   4.1   4.0   3.8   4.4
  9.|-- te-7-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.250)     36.7%    30    19   4.7   2.9   2.4   6.7
 10.|-- c-69-181-142-213.hsd1.ca.comcast.net (69.181.142.213)           40.0%    30    18  13.4  13.1  11.8  15.9
=== END

=== Wed Jun  8 22:05:00 PDT 2016  (1465448700)
Start: Wed Jun  8 22:05:00 2016
HOST: mambo.koitsu.org                                                Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- ???                                                             100.0    30     0   0.0   0.0   0.0   0.0
  2.|-- 45.63.80.1                                                       0.0%    30    30   0.4   0.9   0.3   6.2
  3.|-- xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.89)               0.0%    30    30   0.4   1.0   0.3  16.3
  4.|-- xe-7-0-7.sjc12.ip4.gtt.net (89.149.184.237)                      0.0%    30    30   3.5   1.2   0.6   3.7
  5.|-- as7922.sjc12.ip4.gtt.net (173.205.58.170)                        0.0%    30    30   1.2   1.4   1.0   2.9
  6.|-- hu-0-3-0-5-cr01.9greatoaks.ca.ibone.comcast.net (68.86.83.129)  40.0%    30    18   2.1   2.5   1.3   3.5
  7.|-- be-7922-ar01.hayward.ca.sfba.comcast.net (68.86.94.154)         40.0%    30    18   2.9   2.8   2.5   3.1
  8.|-- hu-0-4-0-1-sur04.santaclara.ca.sfba.comcast.net (68.87.192.254) 43.3%    30    17   3.9   4.0   3.9   4.2
  9.|-- te-7-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.250)     43.3%    30    17   2.8   2.6   2.4   3.3
 10.|-- c-69-181-142-213.hsd1.ca.comcast.net (69.181.142.213)           46.7%    30    16  12.8  13.7  12.3  19.7
=== END

=== Wed Jun  8 22:06:00 PDT 2016  (1465448760)
Start: Wed Jun  8 22:06:00 2016
HOST: mambo.koitsu.org                                                Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- ???                                                             100.0    30     0   0.0   0.0   0.0   0.0
  2.|-- 45.63.80.1                                                       0.0%    30    30   0.8   0.7   0.3   5.3
  3.|-- xe-1-2-3.cr1.sjc2.us.as4436.gtt.net (69.22.130.89)               0.0%    30    30   0.4   0.4   0.3   0.7
  4.|-- xe-7-0-7.sjc12.ip4.gtt.net (89.149.184.237)                      0.0%    30    30   0.7   0.9   0.6   2.5
  5.|-- as7922.sjc12.ip4.gtt.net (173.205.58.170)                        0.0%    30    30   1.2   1.2   1.0   1.7
  6.|-- hu-0-4-0-0-cr01.9greatoaks.ca.ibone.comcast.net (68.86.86.209)  30.0%    30    21   1.5   2.7   1.4   5.7
  7.|-- be-7922-ar01.hayward.ca.sfba.comcast.net (68.86.94.154)         40.0%    30    18   2.7   2.7   2.5   3.1
  8.|-- hu-0-4-0-1-sur04.santaclara.ca.sfba.comcast.net (68.87.192.254) 36.7%    30    19   3.9   3.9   3.9   4.4
  9.|-- te-7-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.250)     40.0%    30    18   2.5   2.5   2.4   2.8
 10.|-- c-69-181-142-213.hsd1.ca.comcast.net (69.181.142.213)           40.0%    30    18  22.0  13.4  12.1  22.0
=== END

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |




More information about the Outages mailing list