[outages] Daily packet loss/routing issue (Layer42/Verizon/Deutsche Telekom/Comcast)

Fri Oct 25 23:49:03 EDT 2013

Here's another one of these incredibly annoying situations that baffles
me (and of course figuring out who's responsible is not something I have
the power to do).

Been watching this one happen every day, usually between the hours of
1700 and 2000 PDT (UTC-0700).  Packet loss and latency lasts anywhere
between 30 to 75 minutes.  From today around 1915:

src IP: 204.8.213.80
dst IP: 76.102.14.35

 Host                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ---                                          0.0%   161   21.5   1.4   0.3  59.3   6.0
 2. ---                                          0.0%   160    0.4   1.2   0.3  41.4   4.6
 3. ---                                          0.0%   160    0.3   2.2   0.2  44.5   6.5
 4. ---                                          0.0%   160    1.5   3.3   1.2  47.2   6.7
 5. xe0-1-0-1.core1.sv8.layer42.net              0.0%   160    2.2   2.1   1.9   2.4   0.1
 6. 193.159.165.73                               0.0%   160    4.9   4.1   1.9   8.1   1.3
 7. 80.156.163.154                              24.5%   160  1000. 933.0   6.3 1738. 624.0
 8. pos-1-12-0-0-cr01.sanjose.ca.ibone.comcast  27.7%   160  987.5 932.7   7.2 1706. 647.7
 9. he-0-6-0-0-ar01.sfsutro.ca.sfba.comcast.ne  26.4%   160  981.4 917.1   7.2 1719. 642.9
10. te-0-4-0-10-ur05.santaclara.ca.sfba.comcas  30.8%   160  980.0 895.1   9.0 1710. 636.9
11. te-6-0-acr03.santaclara.ca.sfba.comcast.ne  17.0%   160  967.3 614.0   6.9 1224. 518.1
12. c-76-102-14-35.hsd1.ca.comcast.net          18.9%   160  970.2 609.1  16.2 1229. 517.8

src IP: 76.102.14.35
dst IP: 204.8.213.80

 Host                                           Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. gw.home.lan                                  0.0%   563   563   0.3   0.2   0.2   1.9
 2. 76.102.12.1                                  0.0%   563   563   8.4   8.8   7.9  22.8
 3. te-0-2-0-5-ur06.santaclara.ca.sfba.comcast.  0.0%   563   563   8.7   8.9   8.1  23.1
 4. te-1-1-0-9-ar01.oakland.ca.sfba.comcast.net  0.0%   563   563  13.5  12.4   9.7  30.2
 5. be-90-ar01.sfsutro.ca.sfba.comcast.net       0.0%   563   563  12.5  13.7  11.1  41.5
 6. he-3-8-0-0-cr01.sanjose.ca.ibone.comcast.ne  0.0%   563   563  13.2  15.0  12.5  25.0
 7. be-14-pe02.11greatoaks.ca.ibone.comcast.net  0.0%   562   562  16.5  16.5  15.6  43.1
 8. 23-30-206-94-static.hfc.comcastbusiness.net  0.0%   562   562  16.4  22.5  15.6  40.2
 9. 0.xe-2-0-8.XL3.SJC7.ALTER.NET                0.7%   562   558  20.3  29.6  15.3 124.7
10. 0.so-4-0-0.XL1.SJC1.ALTER.NET                1.1%   562   556  26.6  22.9  16.8 174.2
    0.so-7-0-0.XL1.SJC1.ALTER.NET
11. POS1-0.XR1.SJC1.ALTER.NET                   23.0%   562   432 196.6 150.0  17.8 459.6
12. 193.ATM7-0.GW4.SJC1.ALTER.NET               24.6%   562   424 1133. 669.3  17.9 1835.
13. ???

What interests me: hops #6 and #7 shown in the first mtr:

193.159.165.73 = AS3320 = Deutsche Telekom AG
80.156.163.154 = AS3320 = Deutsche Telekom AG

Using Layer42's own looking glass long after the issue was over:

lg-west>show ip bgp 76.102.14.35
BGP routing table entry for 76.96.0.0/11, version 4070745
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  Refresh Epoch 1
  8121 3320 7922
    69.36.239.8 from 69.36.239.8 (69.36.239.8)
      Origin IGP, metric 301, localpref 100, valid, external, best
      rx pathid: 0, tx pathid: 0x0

lg-west>traceroute 76.102.14.35
Type escape sequence to abort.
Tracing the route to c-76-102-14-35.hsd1.ca.comcast.net (76.102.14.35)
VRF info: (vrf in name/id, vrf out name/id)
  1 69.36.229.177 [AS 8121] 1 msec 1 msec 1 msec
  2 xe0-0-0-4.core1.sv8.layer42.net (65.50.198.161) [AS 8121] 3 msec 2 msec 2 msec
  3 193.159.165.73 [AS 3320] 4 msec 2 msec 3 msec
  4 80.156.163.154 [AS 3320] 4 msec 5 msec 3 msec
  5 pos-1-12-0-0-cr01.sanjose.ca.ibone.comcast.net (68.86.87.141) [AS 7922] 7 msec 6 msec 4 msec
  6 he-0-5-0-0-ar01.sfsutro.ca.sfba.comcast.net (68.86.91.46) [AS 7922] 6 msec 5 msec 8 msec
  7 te-0-4-0-6-ur05.santaclara.ca.sfba.comcast.net (69.139.198.169) [AS 7922] 5 msec 6 msec 7 msec
  8 te-6-0-acr03.santaclara.ca.sfba.comcast.net (68.86.249.66) [AS 7922] 5 msec 6 msec 5 msec
  9 c-76-102-14-35.hsd1.ca.comcast.net (76.102.14.35) [AS 7922] 14 msec 18 msec 14 msec

Two final things to note:

1. When the packet loss/latency issue is not occurring (e.g. presently),
the paths are still the same.  If packets were actually going from
Sunnyvale (Layer42) to Germany I'd expect a lot higher than 4ms-7ms
latency consistently,

2. The source/destination of 204.8.213.80 is a box at my place of work;
border of network intentionally filters.  We do peer with Layer42
directly, but I have not pursued this with them yet (as said initially,
it's hard for me to accurately point fingers).

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Making life hard for others since 1977.             PGP 4BD6C0CB |