[outages] Twitter connectivity issues?

Fri Jan 11 10:12:13 EST 2013

Hi list,

I tried to send this yesterday, but after 15 hours I don't see my post, so appologies for any duplicates. 

Customer complaints started rolling in around 11:30 AM EST (GMT-0:500) yesterday, or approx 23 hours ago. I'm seeing what appears to be a broken Layer 3 or 4 port channel (load balancing) somewhere in Twitter land. I have no idea if this is affecting anyone other than us as searching hasn't yielded anything, and Twitter's status page shows no issues. 

At the end of my message I've put some example of what I'm seeing. Behaviour changes based on source/dest IP, based on many customer and infrastructure addresses in different subnets, and a variety of the A records returned for twitter.com. We are multihomed so that might obscure the problem, I've included a sample out our 2nd provider. 

The problem appears to be somewhere past ae60.pao1-cr1.twttr.com (199.16.159.85)

If anyone from Twitter is out there please drop me a line... I've dropped a line to the ARIN NOC contact but haven't gotten a response. We are AS 23252

If anyone else is experiencing this I'd love to know also

Thanks
---
Ross Halliday
Network Operations
WTC Communications

Office: 613-547-6939 x203
Helpdesk: 866-547-6939 option 2
http://www.wtccommunications.ca

edge-wprt-c6509#traceroute 199.59.150.39 source 66.102.70.237

Type escape sequence to abort. 
Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39)

  1 ge-6-1-117.car1.Toronto2.Level3.net (4.59.180.17) [AS 3356] 4 msec 4 msec 4 msec
  2 4.69.153.77 [AS 3356] [MPLS: Label 493184 Exp 0] 4 msec 4 msec 4 msec
  3 ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242) [AS 3356] [MPLS: Label 492736 Exp 0] 4 msec 8 msec 4 msec
  4 ae-7-1.ebr2.NewYork1.Level3.net (4.69.151.114) [AS 3356] [MPLS: Label 1494 Exp 0] 24 msec 24 msec 20 msec
  5 ae-6-6.ebr2.NewYork2.Level3.net (4.69.141.22) [AS 3356] [MPLS: Label 1343 Exp 0] 16 msec 20 msec 16 msec
  6 ae-1-100.ebr1.NewYork2.Level3.net (4.69.135.253) [AS 3356] [MPLS: Label 1278 Exp 0] 20 msec 24 msec 24 msec
  7 ae-3-3.ebr2.Washington1.Level3.net (4.69.132.89) [AS 3356] [MPLS: Label 1355 Exp 0] 24 msec 24 msec 20 msec
  8 ae-82-82.csw3.Washington1.Level3.net (4.69.134.154) [AS 3356] [MPLS: Label 1452 Exp 0] 24 msec
    ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) [AS 3356] [MPLS: Label 1390 Exp 0] 24 msec
    ae-62-62.csw1.Washington1.Level3.net (4.69.134.146) [AS 3356] [MPLS: Label 1388 Exp 0] 24 msec
  9 ae-2-70.edge2.Washington4.Level3.net (4.69.149.80) [AS 3356] 24 msec 24 msec
    ae-1-60.edge2.Washington4.Level3.net (4.69.149.16) [AS 3356] 36 msec
 10 TWITTER-INC.edge2.Washington4.Level3.net (4.53.114.102) [AS 3356] 20 msec 20 msec 20 msec
 11 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 96 msec 92 msec 92 msec
 12  *  *  *
 13 r-199-59-150-39.twttr.com (199.59.150.39) [AS 13414] 92 msec 100 msec 100 msec
edge-wprt-c6509#traceroute 199.59.150.39 source 66.102.70.233

Type escape sequence to abort. 
Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39)

  1 ge-6-1-117.car1.Toronto2.Level3.net (4.59.180.17) [AS 3356] 4 msec 4 msec 4 msec
  2 4.69.153.77 [AS 3356] [MPLS: Label 493184 Exp 0] 28 msec 32 msec 4 msec
  3 ae-0-11.bar2.Toronto1.Level3.net (4.69.151.242) [AS 3356] [MPLS: Label 492736 Exp 0] 16 msec 36 msec 4 msec
  4 ae-7-1.ebr2.NewYork1.Level3.net (4.69.151.114) [AS 3356] [MPLS: Label 1494 Exp 0] 24 msec 24 msec 24 msec
  5 ae-6-6.ebr2.NewYork2.Level3.net (4.69.141.22) [AS 3356] [MPLS: Label 1343 Exp 0] 20 msec 16 msec 20 msec
  6 ae-1-100.ebr1.NewYork2.Level3.net (4.69.135.253) [AS 3356] [MPLS: Label 1278 Exp 0] 16 msec 20 msec 16 msec
  7 ae-3-3.ebr2.Washington1.Level3.net (4.69.132.89) [AS 3356] [MPLS: Label 1355 Exp 0] 20 msec 24 msec 24 msec
  8 ae-92-92.csw4.Washington1.Level3.net (4.69.134.158) [AS 3356] [MPLS: Label 1396 Exp 0] 28 msec
    ae-72-72.csw2.Washington1.Level3.net (4.69.134.150) [AS 3356] [MPLS: Label 1390 Exp 0] 28 msec 24 msec
  9 ae-4-90.edge2.Washington4.Level3.net (4.69.149.208) [AS 3356] 24 msec
    ae-2-70.edge2.Washington4.Level3.net (4.69.149.80) [AS 3356] 92 msec
    ae-3-80.edge2.Washington4.Level3.net (4.69.149.144) [AS 3356] 24 msec
 10 TWITTER-INC.edge2.Washington4.Level3.net (4.53.114.102) [AS 3356] 20 msec 20 msec 24 msec
 11 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 92 msec 92 msec 92 msec
 12  *  *  *
 13  *  *  *
 14  *  *  *
 15  * 

edge-wprt-c6509#traceroute 199.59.150.39

Type escape sequence to abort. 
Tracing the route to r-199-59-150-39.twttr.com (199.59.150.39)

  1 ge5-1-5.core1.toronto1.nexicom.net (76.75.127.145) [AS 11666] 0 msec 8 msec 4 msec
  2 xe2-2-0-1.core1.toronto1.nexicom.net (98.124.49.233) [AS 32881] 4 msec 4 msec 4 msec
  3 206.126.236.97 [AS 8218] 20 msec 20 msec 20 msec
  4 ae60.pao1-cr1.twttr.com (199.16.159.85) [AS 13414] 80 msec 80 msec 88 msec
  5  *  *  *
  6  *  *  *
  7  *  *  *
  8  *  *  *
  9  *  *  *
 10  *  *  *
 11  *  *  *
 12  *  *  *
 13  *  *  *
 14  *  *  *