[outages] Facebook is down

[Frank Bulk]

My previously posted IPv4 and IPv6 traceroutes show a 4 msec incremental hop
between atl1 and frc1, too.  If the host names are I accurate, I admit, too,
that I don't understand how one can get between Atlanta and San Francisco in
~4 msec.

My daily IPv6 tcptraceroute6 snapshot shows that I normally get to
2a03:2880:2110:3f01:face:b00c:: via that ATL hop, but Chicago to Atlanta
taking almost 30 msec?  More likely some asymmetric routing going on here.
<snip>
 7  sl-st31-chi-bu-20.v6.sprintlink.net (2600:0:2:1239:144:232:1:104)
16.012 ms  15.678 ms  15.624 ms
 8  2600:4:2000::f (2600:4:2000::f)  15.428 ms  15.376 ms  15.396 ms
 9  ae1.bb02.ord1.tfbnw.net (2620:0:1cff:dead:beee::452)  15.704 ms  15.603
ms  15.625 ms
10  ae12.bb02.atl1.tfbnw.net (2620:0:1cff:dead:beef::1336)  43.805 ms
36.058 ms  48.112 ms
11  ae11.bb04.frc1.tfbnw.net (2620:0:1cff:dead:beef::3be)  40.169 ms  40.171
ms  40.120 ms
12  ae40.dr02.frc1.tfbnw.net (2620:0:1cff:dead:beef::1c7)  39.875 ms  39.934
ms  39.774 ms
13  po1020.csw03d.frc1.tfbnw.net (2620:0:1cff:dead:beef::163f)  40.388 ms
40.266 ms  40.973 ms
14  * * *
15  www6-slb-ecmp-03-frc1.facebook.com (2a03:2880:2110:3f01:face:b00c::)
40.239 ms [open]  40.160 ms  40.212 ms

Frank

-----Original Message-----
From: Jeremy Chadwick [mailto:jdc at koitsu.org] 
Sent: Monday, January 28, 2013 8:17 PM
To: Frank Bulk
Cc: 'Josh Luthman'; outages at outages.org
Subject: Re: [outages] Facebook is down

On Mon, Jan 28, 2013 at 07:44:13PM -0600, Frank Bulk wrote:
> Appears to be inside of fb's network - prn1 = Prineville and San Francisco
=
> frc1.  {snip}

$ mtr 173.252.100.16

 Host                                           Loss%   Snt   Rcv  Last
Avg  Best  Wrst
 1. gw.home.lan                                  0.0%     7     7   0.3
0.3   0.2   0.4
 2. c-67-180-84-1.hsd1.ca.comcast.net            0.0%     6     6  21.9
23.8  21.9  27.0
 3. te-0-0-0-12-ur05.santaclara.ca.sfba.comcast  0.0%     6     6  10.1
9.9   8.5  11.5
 4. te-1-1-0-4-ar01.sfsutro.ca.sfba.comcast.net  0.0%     6     6  12.7
15.8  12.7  20.7
 5. he-1-5-0-0-cr01.sanjose.ca.ibone.comcast.ne  0.0%     6     6  18.5
18.9  14.6  21.1
 6. pos-2-11-0-0-cr01.losangeles.ca.ibone.comca  0.0%     6     6  22.1
22.4  20.7  24.0
 7. xe-0-0-1-0-pe01.onewilshire.ca.ibone.comcas  0.0%     6     6  21.0
21.3  20.9  21.6
 8. 173.167.59.58                                0.0%     6     6  20.7
21.9  20.1  25.0
 9. ae1.bb02.lax1.tfbnw.net                      0.0%     6     6  20.9
21.6  20.4  23.4
10. ae13.bb02.atl1.tfbnw.net                     0.0%     6     6  84.1
87.5  82.3  99.5
11. ae7.bb02.frc1.tfbnw.net                      0.0%     6     6  87.7
90.2  87.1 103.0
12. ae2.dr03.frc1.tfbnw.net                      0.0%     6     6  83.5
83.8  83.5  85.2
13. ???

To me that looks like the forward path is:

 1. LAN
 2. Comcast: Mountain View, CA
 3. Comcast: Santa Clara, CA
 4. Comcast: San Francisco, CA (always been an assumption on my part)
 5. Comcast: San Jose
 6. Comcast: Los Angeles
 7. Comcast: Los Angeles (One Wilshire)
 8. Comcast: possible FB<->Comcast peering point
 9. Facebook: Los Angeles
10. Facebook: Atlanta (note 60ms+ latency jump for cross-country)
11. Facebook: ???
11. Facebook: ???

If it went from LAX->ATL->SFO, I would expect to see the latency jump
~60-80ms at #9/#10, and then again at #10/#11.  But again, this is the
forward-path only.  And I refuse to believe FRC means Franca Airport in
Brazil.  ;-)

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |