[outages] Comcast<->AT&T packet loss (possibly within CA)

[Jeremy Chadwick]

Since roughly Friday, I've been seeing what appears to be packet loss
somewhere within Comcast/AT&T network mesh.  Source and destination IPs
are provided below as well, ditto with some mtrs from src->dst and
dst->src.  I keep periodic mtrs (both directions) going all the way back
to 03/04.  I can make all of those logs available if asked.

The issue started on 03/07 @ 21:33 PST suddenly -- not a "gradual"
increase -- and lasted until an undetermined time (very hard to tell
from mtrs) but I'd estimate ~02:00 PST on 03/08 (today).

The issue then appeared to start back up again ~07:00 PST, though it's
hard to give an exact time (seems sort of a gradual increase, thus hard
to pinpoint).  It's been ongoing since.

The loss varies from 3% to 20%, but you can definitely "feel" it across
an SSH session, so it's not ICMP prio.

I will make myself clear: it's very hard to "show" someone the way this
problem manifests itself, because the packet loss will vary all over the
place between different hops.  It *definitely* starts at a particular
point and "trickles down", but due to the fact that the loss is a
smaller percentage, there are times where a hop will suddenly show 0%.
TL;DR -- You'd really have to see a longer log (say, an hour's worth) to
be able to say "ah yes, this really is a problem" and not blow it off as
ICMP prio.

And as usual, there's one of those "mystery routers" (hop #8 in the
first example) that peering providers looooooove to use as a scapegoat
when it comes to shifting blame, ex. provider A says "that's a device
owned by provider B", provider B says "that device is provider A's
responsibility", and neither side does anything about the issue.
However I should note that the "mystery router" usually does show some
degree of loss even when this issue isn't occurring (likely ICMP prio on
the device), but that makes it even more difficult to determine where
the issue begins.


src IP: 76.102.14.35   (Comcast; Mountain View, CA)
dst IP: 204.109.61.174 (RootBSD; Dallas, TX)

=== Sat Mar  8 16:22:00 PST 2014  (1394324520)
Start: Sat Mar  8 16:22:00 2014
HOST: icarus.home.lan                                                 Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- gw.home.lan (192.168.1.1)                                        0.0%    30    30   0.4   0.3   0.2   0.4
  2.|-- 76.102.12.1                                                      0.0%    30    30   8.0   8.8   8.0  12.2
  3.|-- te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.net (68.86.249.253)   0.0%    30    30   8.2   9.0   8.2  16.5
  4.|-- te-1-1-0-1-ar01.oakland.ca.sfba.comcast.net (69.139.198.94)      0.0%    30    30  11.9  12.2  10.1  15.0
  5.|-- be-90-ar01.sfsutro.ca.sfba.comcast.net (68.85.155.14)            0.0%    30    30  12.0  12.5  10.1  15.1
  6.|-- he-3-8-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.94.85)       0.0%    30    30  13.0  14.0  11.8  18.0
  7.|-- pos-0-3-0-0-pe01.11greatoaks.ca.ibone.comcast.net (68.86.87.18)  0.0%    30    30  15.6  17.2  15.3  19.8
  8.|-- 192.205.37.1                                                    70.0%    30     9  54.8  67.7  53.6 102.2
  9.|-- cr2.sffca.ip.att.net (12.122.86.202)                            13.3%    30    26  65.2  63.5  61.0  65.7
 10.|-- cr2.la2ca.ip.att.net (12.122.31.133)                             6.7%    30    28  63.3  63.5  60.9  75.2
 11.|-- cr2.dlstx.ip.att.net (12.122.28.177)                             3.3%    30    29  65.2  63.7  61.1  69.7
 12.|-- ggr6.dlstx.ip.att.net (12.122.138.113)                           6.7%    30    28  60.3  64.5  59.9 153.6
 13.|-- 12.90.228.14                                                     6.7%    30    28  60.3  60.7  60.2  62.5
 14.|-- border1.pc1-bbnet1.dal004.pnap.net (216.52.191.19)               3.3%    30    29  60.3  60.2  59.8  60.5
 15.|-- giglinx-60.border1.dal004.pnap.net (216.52.189.46)               3.3%    30    29  59.9  60.2  59.8  61.4
 16.|-- 204.109.62.46                                                    6.7%    30    28  60.1  60.5  60.1  62.7
 17.|-- mambo.koitsu.org (204.109.61.174)                                3.3%    30    29  60.7  60.9  60.1  63.4
=== END


src IP: 204.109.61.174 (RootBSD; Dallas, TX)
dst IP: 76.102.14.35   (Comcast; Mountain View, CA)

=== Sat Mar  8 16:22:00 PST 2014  (1394324520)
Start: Sat Mar  8 16:22:00 2014
HOST: mambo.koitsu.org                                                Loss%   Snt   Rcv  Last   Avg  Best  Wrst
  1.|-- 204.109.61.173                                                   0.0%    30    30   0.5   1.3   0.4  15.0
  2.|-- 204.109.62.45                                                    0.0%    30    30   0.5   0.5   0.3   1.2
  3.|-- border1.ge1-6.giglinx-60.dal004.pnap.net (216.52.189.45)         0.0%    30    30   0.5   0.6   0.4   4.7
  4.|-- core3.pc1-bbnet1.ext1a.dal.pnap.net (216.52.191.41)              0.0%    30    30   0.9   1.0   0.9   1.2
  5.|-- dax-edge-03.inet.qwest.net (67.133.189.93)                       0.0%    30    30   0.6   2.0   0.5  22.8
  6.|-- 63-235-82-234.dia.static.qwest.net (63.235.82.234)               0.0%    30    30   1.4   1.3   1.0   1.7
  7.|-- be-13-cr01.dallas.tx.ibone.comcast.net (68.86.82.141)            0.0%    30    30   1.3   2.7   1.0   4.9
  8.|-- he-0-14-0-0-cr01.losangeles.ca.ibone.comcast.net (68.86.85.141)  0.0%    30    30  35.6  33.6  31.8  35.7
  9.|-- he-1-8-0-0-ar01.oakland.ca.sfba.comcast.net (68.86.89.54)        3.3%    30    29  52.8  53.5  51.5  55.5
 10.|-- te-0-4-0-5-ur06.santaclara.ca.sfba.comcast.net (68.86.143.97)    0.0%    30    30  52.1  52.2  51.9  52.3
 11.|-- te-6-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.66)       6.7%    30    28  53.0  53.0  52.8  53.8
 12.|-- c-76-102-14-35.hsd1.ca.comcast.net (76.102.14.35)                3.3%    30    29  60.4  60.5  59.9  63.8
=== END

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |