[outages] AT&T SFO to Twitter -- possible routing issue or BGP hijack

[Jeremy Chadwick]

Found twitter.com was not loading tonight.  Dug in.

$ host www.twitter.com
www.twitter.com is an alias for twitter.com.
twitter.com has address 104.244.42.129
twitter.com mail is handled by 30 ASPMX2.GOOGLEMAIL.com.
twitter.com mail is handled by 20 alt2.aspmx.l.google.com.
twitter.com mail is handled by 20 alt1.aspmx.l.google.com.
twitter.com mail is handled by 10 aspmx.l.google.com.
twitter.com mail is handled by 30 ASPMX3.GOOGLEMAIL.com.

$ dig ns twitter.com +short
b.r06.twtrdns.net.
a.r06.twtrdns.net.
c.r06.twtrdns.net.
d.r06.twtrdns.net.
a.u06.twtrdns.net.
b.u06.twtrdns.net.
c.u06.twtrdns.net.
d.u06.twtrdns.net.

$ dig @b.r06.twtrdns.net a twitter.com +short
104.244.42.129

And 104.244.42.129 does indeed point to Twitter (per WHOIS/ARIN), so doesn't
appear to be a DNS-related thing.  Onward we go:

$ mtr www.twitter.com
                                                                      Packets               Pings
 Host                                                               Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. 192.168.1.254                                                    0.0%     9     9   0.6   0.6   0.5   0.8
 2. 172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.232.1)      0.0%     9     9   2.3   2.3   1.5   4.3
 3. 71.148.149.42 (71.148.149.42)                                    0.0%     8     8   2.6   2.7   1.7   3.9
 4. 12.242.117.22 (12.242.117.22)                                    0.0%     8     8   4.6   6.1   3.6   8.2
 5. att-gw.sfo.pccw.net (192.205.32.82)                              0.0%     8     8   6.7   6.5   5.3   8.1
 6. Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30)             0.0%     8     8 121.6 121.1 119.6 122.8
 7. 63-222-51-222.static.pccwglobal.net (63.222.51.222)              0.0%     8     8 154.3 154.4 153.3 155.8
 8. (waiting for reply)
 9. 104.244.42.129 (104.244.42.129)                                  0.0%     8     8 151.5 152.6 151.4 153.8

$ mtr -z www.twitter.com

                                                                      Packets               Pings
 Host                                                               Loss%   Snt   Rcv  Last   Avg  Best  Wrst
 1. AS???    192.168.1.254                                           0.0%     8     8   0.8   0.6   0.3   0.8
 2. AS7018   172-10-232-1.lightspeed.sntcca.sbcglobal.net (172.10.2  0.0%     8     8   1.3   2.2   1.0   4.1
 3. AS7018   71.148.149.42 (71.148.149.42)                           0.0%     8     8   2.2   3.4   1.9   5.0
 4. AS7018   12.242.117.22 (12.242.117.22)                           0.0%     8     8   6.0   6.1   4.5   7.6
 5. AS7018   att-gw.sfo.pccw.net (192.205.32.82)                     0.0%     8     8   5.6   6.2   4.8   7.7
 6. AS3491   Bundle-Ether45.br04.osa01.pccwbtn.net (63.223.26.30)    0.0%     8     8 120.4 120.7 120.0 121.5
 7. AS3491   63-222-51-222.static.pccwglobal.net (63.222.51.222)     0.0%     8     8 154.1 155.4 153.6 161.5
 8. (waiting for reply)
 9. AS13414  104.244.42.129 (104.244.42.129)                         0.0%     7     7 152.7 152.2 150.8 154.0

AS3491 (pccwbtn.net) is PCCW Global, though a WHOIS on pccwbtn.net says
they're PCCW-HKT out of Hong Kong, which would explains the huge jump in
latency (6ms -> 121ms) since I'm located in California.  63.223.26.30 is
also PCCW Global.

PeeringDB says https://www.peeringdb.com/net/674 (AT&T) has a looking
glass server at http://route-server.ip.att.net/ but the webserver is not
listening on TCP port 80, nor 443:

$ telnet route-server.ip.att.net 80
Trying 12.0.1.28...
telnet: connect to address 12.0.1.28: Connection refused
telnet: Unable to connect to remote host
$ telnet route-server.ip.att.net 443
Trying 12.0.1.28...
telnet: connect to address 12.0.1.28: Connection refused
telnet: Unable to connect to remote host

And for those that want source and destinations:

src IP: 107.197.104.143 (AT&T Fibre)
dst IP: 104.244.42.129  (Twitter)

-- 
| Jeremy Chadwick                              jdc_at_koitsu.org |
| UNIX Systems Administrator                      PGP 0x2A389531 |
| Making life hard for others since 1977.                        |