[Outages-discussion] [outages] Facebook high packet loss & latency

[Jeremy Chadwick]

I briefly poked at this when I saw the report and chose to say nothing;
I don't use Facebook, I only looked out of curiosity.

Issue appeared specific to 157.240.3.20 (edge-star-shv-01-sea1).

www.messenger.com is a CNAME to star.facebook.com.  star.facebook.com
returns an A record with a short TTL (45 seconds), and the record
returned varies**.  It's safe to say "sea1" refers to Seattle, which
also explains the RTT shown in the traceroute (MV->SEA).

When I analysed the situation with mtr, I saw no packet loss in the path
that couldn't otherwise be explained by ICMP prioritisation -- only the
final hop/destination showed loss (up to 45%).  But due to the
likelihood of asymmetric routing***, I can only see half the picture
(read: I have no way of knowing if the loss happens on the return path).

If you were lucky enough to get 157.240.22.19 (edge-star-shv-01-sjc3)
out of San Jose, you'd find it was reliable -- no packet loss.


** - "Varies" doesn't mean RR DNS.  It's a DNS-based LB model where the
A record returned changes (quite common).  Maybe it's GeoIP-based, maybe
it's not, maybe it's based on Magical Sauce That Only Facebook Knows Or
Is Willing To Disclose.  No way to know because I didn't take snapshots
of the DNS SOA at the time; the serials aren't RFC1912-compliant so
they're just incrementing numbers, not ISO-8601-like.

*** - I use any opportunity possible to force others into reading this
highly applicable NANOG presentation given back in late 2009:
https://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf

-- 
| Jeremy Chadwick                                 jdc at koitsu.org |
| UNIX Systems Administrator                      PGP 0x2A389531 |
| Making life hard for others since 1977.                        |

On Tue, Nov 20, 2018 at 12:42:56AM +0000, Donn Lasher via Outages wrote:
> Not to +me.too, but downdetector was lit up last night, noted it personally from multiple points on the west coast. Never found an RCA/RFO, but something was really borked for a bit.
> 
> 
> On 11/18/18, 9:15 PM, "Outages on behalf of Kevin Chen via Outages" <outages-bounces at outages.org on behalf of outages at outages.org> wrote:
> 
>     EXTERNAL MAIL: outages-bounces at outages.org
>     
>     Seeing high packet loss and latency to Facebook services from Comcast in Mountain View, CA.
>     
>     * www.facebook.com
>     * www.instagram.com
>     * www.messenger.com
>     
>     Ping for www.messenger.com:
>     
>     --- star.c10r.facebook.com ping statistics ---
>     164 packets transmitted, 110 packets received, 32.9% packet loss
>     round-trip min/avg/max/stddev = 48.484/56.295/261.958/27.756 ms
>     
>     And here is a traceroute:
>     
>     $ traceroute www.messenger.com
>     traceroute to star.c10r.facebook.com (157.240.3.20), 64 hops max, 52 byte packets
>      1  172.16.0.1 (172.16.0.1)  4.061 ms  5.813 ms  8.769 ms
>      2  96.120.91.25 (96.120.91.25)  24.801 ms  20.402 ms  19.775 ms
>      3  be-20009-rur02.santaclara.ca.sfba.comcast.net (68.87.195.189)  14.825 ms  17.473 ms  15.166 ms
>      4  162.151.78.129 (162.151.78.129)  17.093 ms  15.958 ms  19.341 ms
>      5  162.151.78.253 (162.151.78.253)  17.292 ms  18.793 ms  18.504 ms
>      6  be-3651-cr02.sunnyvale.ca.ibone.comcast.net (68.86.91.73)  18.208 ms  15.616 ms  20.786 ms
>      7  be-11025-cr01.9greatoaks.ca.ibone.comcast.net (68.86.87.158)  23.369 ms  19.778 ms  20.170 ms
>      8  be-10825-cr01.seattle.wa.ibone.comcast.net (68.86.85.197)  39.775 ms  38.383 ms  44.532 ms
>      9  be-10847-pe02.seattle.wa.ibone.comcast.net (68.86.86.226)  40.979 ms  39.799 ms  39.582 ms
>     10  * 66.208.228.2 (66.208.228.2)  53.482 ms *
>     11  po103.psw04.sea1.tfbnw.net (173.252.65.197)  50.868 ms
>         po103.psw03.sea1.tfbnw.net (173.252.65.145)  35.061 ms  40.036 ms
>     12  173.252.67.73 (173.252.67.73)  34.965 ms
>         173.252.67.59 (173.252.67.59)  44.327 ms
>         173.252.67.135 (173.252.67.135)  36.749 ms
>     13  edge-star-shv-01-sea1.facebook.com (157.240.3.20)  45.418 ms  36.913 ms  36.493 ms
>     _______________________________________________
>     Outages mailing list
>     Outages at outages.org
>     https://puck.nether.net/mailman/listinfo/outages
>     
>     
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages