[Outages-discussion] [outages] Fwd: RE: FYI Netflix is down

[virendra rode]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 06/30/2012 07:46 AM, Bill Wichers wrote:
> Does anyone know what actually caused the failure? I know there was
> a "power outage" due to weather, but presumably the physical site
> would have also had generator backup so that must have also
> failed?
> 
> [Sent using Blackberry Messaging]
- -------------------------
Nothing that I've seen yet. I for one am looking forward to lesson
learned summary from EC2 team and will post it to wiki page as it
comes available.

I'm curious as to why wouldn't you build failover within your system
as opposed to relying on (v)cloud solution. If your business
availability is so important then you need to build around common
failures. I don't have any services hanging off AWS cloud so I can't
speak w/ surety but load balancing your application across multiple
availability zone could have provided uptime by distributing across
multiple zones and /or data center, no?

I'm going on a limb here but I don't think amazon shoulders all the
blame in fact the same network engineers who point fingers at amazon
about their service availability should be questioned how a failure in
one data center affected chain of service instead of using high
availability practices. Maybe too high of a cost for redundancy?

Just curious.

regards,
/virendra

> 
> ----- Original Message ----- From:
> outages-discussion-bounces at outages.org
> <outages-discussion-bounces at outages.org> To: Paul Ferguson
> <fergdawgster at gmail.com> Cc: outages-discussion at outages.org
> <outages-discussion at outages.org> Sent: Sat Jun 30 01:46:05 2012 
> Subject: Re: [Outages-discussion] [outages] Fwd: RE: FYI Netflix is
> down
> 
> That is a good point.  Why didn't they survive in an adjacent AZ
> in Ashburn?  Reports say only a single AZ lost power.
> 
> On Fri, Jun 29, 2012 at 10:15 PM, Paul Ferguson
> <fergdawgster at gmail.com> wrote:
>> I'm looking forward to additional details on this outage --
>> Netflix has previously avoided EC2 outages due to their creative
>> use of Chaos Monkey:
>> 
>> http://techblog.netflix.com/2011/07/netflix-simian-army.html
>> 
>> FYI,
>> 
>> - ferg
>> 
>> On Fri, Jun 29, 2012 at 10:07 PM, Kevin Blackham
>> <blackham at gmail.com> wrote:
>> 
>>> On Fri, Jun 29, 2012 at 9:40 PM, Paul Ferguson
>>> <fergdawgster at gmail.com> wrote:
>>>> http://venturebeat.com/2012/06/29/amazon-outage-netflix-instagram-pinterest/
>>>
>>>
>>>> 
"The outage underscores the vulnerabilities of depending on the public
>>> cloud versus using your own data centers."
>>> 
>>> Let me fix that: "The outage underscores the lack of attention
>>> to failure tolerance by many affected services." Building my
>>> own sites isn't going to keep them from failing. Every
>>> datacenter will fail eventually.
>>> 
>>> Granted, when an Amazon AZ fails, history has shown the thrash
>>> induced by recovery mechanisms can also take out adjacent AZs
>>> (e.g. network overloads vs. EBS), but that doesn't stop an
>>> architect from considering regional redundancy. 
>>> _______________________________________________ 
>>> Outages-discussion mailing list Outages-discussion at outages.org 
>>> https://puck.nether.net/mailman/listinfo/outages-discussion
>> 
>> 
>> 
>> -- "Fergie", a.k.a. Paul Ferguson fergdawgster(at)gmail.com
> _______________________________________________ Outages-discussion
> mailing list Outages-discussion at outages.org 
> https://puck.nether.net/mailman/listinfo/outages-discussion
> 
> 
> 
> _______________________________________________ Outages-discussion
> mailing list Outages-discussion at outages.org 
> https://puck.nether.net/mailman/listinfo/outages-discussion
> 


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iF4EAREIAAYFAk/v4uwACgkQ3HuimOHfh+Hf+AD8DhBin6YV32l9dQBevYnhCURD
PUaMTUMDAXXrBn6+wg4A/0TCBSuzD5Isl2OIZaXTVt+ZNGF2BlJHMFtO/voLbC5q
=p7EO
-----END PGP SIGNATURE-----