[outages] FB down?

[Keegan.Holley at sungard.com]

By webserver I meant the box not the daemon running on it.  I wasn't 
trying to give everyone a lesson on TCP, I was just implying that the 
original post about the problem being based on a location were probably 
false.  I'm still curious if it was the entire site or just a few of the 
servers in a load-balanced pool







Re: [outages] FB down?

Jeremy Chadwick 
to:
outages
04/23/10 07:48 PM


Sent by:
<outages-bounces at outages.org>







On Fri, Apr 23, 2010 at 07:08:14PM -0400, Keegan.Holley at sungard.com wrote:
> Correct me if I'm wrong but aren't RST's sent by the TCP stack in the 
> webserver?  If you are recieving them it would imply that the network 
> connection is fine, but you were sent to a server that is having issues. 

> WIth the amount of traffic they recieve I would assume they use some 
sort 
> of load balancing method.  Try clearing your DNS cache and closing your 
> browser and trying again.

RSTs are sent by the OS (not the "webserver"; there is no TCP stack
inside a webserver) when the underlying userland application calls
close(2) on the descriptor (socket), the process exits, or otherwise
aborts/crashes in some way thus freeing the descriptor that handles the
network I/O.  Facebook's predominantly using Linux[1] which is why I
describe the above model.

Given Facebook's complexity, who knows what the problem was.  Load
balancer or layer 7 filter/re-writer (think F5) issues?  Back-end server
problems?  Software misconfiguration?  Doing burn-outs in an AMC Pacer
by sticking with PHP[2]?  Some developer deciding to just roll something
out in the middle of the day (as is quite common with social networking
sites these days)?  We'll probably never know.

If I had a nickel for every time I've seen FB's environment behave
erratically (easy to confirm with curl), close sockets abruptly, time
out during HTTP transactions, I'd own a series of small islands by now.

[1]: http://www.infoq.com/presentations/Facebook-Software-Stack 
[2]: http://developers.facebook.com/blog/post/358

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |
> 
> 
> 
> 
> 
> 
> 
> [outages] FB down?
> 
> Franck Martin 
> to:
> outages
> 04/23/10 06:12 PM
> 
> 
> Sent by:
> <outages-bounces at outages.org>
> 
> 
> 
> 
> 
> 
> 
> FB is down from Fiji: the connection to the server was reset
> 
> Franck Martin
> http://www.avonsys.com/
> http://www.facebook.com/Avonsys
> twitter: FranckMartin Avonsys
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 


> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20100423/a9ac7f51/attachment.htm>