[Outages-discussion] FW: [outages-discussion] Fastly CDN down?

[Mike Castoro]

Yea, I was being facetious when I said "who knew half the internet used this same CDN".  It's been pretty clear from following Outages mailing list for 7+ years that the entire internet backbone is tied together a little too tightly.

Good links, thanks for sharing.

And yes, props to Fastly engineers for turning around a resolution so quickly.  Unless it was something silly and easy to fix.  

Also, bad timing for Fastly as their stock just popped yesterday.

Mike Castoro | VP of IT
Adrianna Papell Group | 646-723-3885
Chat with me in Teams

-----Original Message-----
From: Outages-discussion <outages-discussion-bounces at outages.org> On Behalf Of Jeremy Chadwick
Sent: Tuesday, June 8, 2021 9:06 AM
To: outages-discussion at outages.org
Subject: Re: [Outages-discussion] FW: [outages-discussion] Fastly CDN down?

External Email!


https://news.ycombinator.com/item?id=27432408 has some discussion about why certain parts of Amazon (not AWS!) uses Fastly, and what Amazon did to mitigate (switched to Cloudfront).

As for "who knew half the internet used this same CDN", I've known this for a while.  The number of companies doing this continue to grow.  For some reason online service providers have still not learned to avoid putting all their eggs in one basket; they will spend exorbitant amounts of time and money to guarantee 5-9s but then introduce critical SPoFs *by choice*.  Another recent example of this same mindset was the AWS Kinesis outage in November, where AWS simply trusted their own tech too
much: https://aws.amazon.com/message/11201/

{off-topic}  There is a pervasive mentality in the online web-based service industry that, put roughly, amounts to "Big Company X uses Thingus, so we should use Thingus too".  This applies to not just CDNs or other service providers, but software as well.  Us old codgers know this isn't a good idea (or if it's deemed acceptable, that the downsides are well-understood and other mitigation methods are available at time of infrastructure design).  "Why do it ourselves when Derpaherpaderp.io can do it for us?" is a dangerous question, and one that should be approached with extreme caution.  Proper monitoring is almost always neglected as well (Ops are told there are more important things to address, rinse lather repeat for years -- a backlog that never gets worked on).  {/off-topic}

That said, ~1-1.5 hours from start to fix is pretty good:
https://status.fastly.com/#past-incidents .  Respects to those in Fastly Operations who likely got woken up + had to deal with it.

--
| Jeremy Chadwick                                 jdc at koitsu.org |
| UNIX Systems Administrator                      PGP 0x2A389531 |
| Making life hard for others since 1977.                        |

On Tue, Jun 08, 2021 at 11:36:24AM +0000, Mike Castoro wrote:
> Added to Outages-Discussion.  Who knew that half the internet used the same CDN edge provider?   Why would AWS and GCP use them?  Our Shopify sites were down this morning and Shopify support scrambled to even know it was happening (plus their status page was down as well).
>
> Mike Castoro | VP of IT
> Adrianna Papell Group | 646-723-3885
> Chat with me in Teams
>
> -----Original Message-----
> From: Outages <outages-bounces at outages.org> On Behalf Of Stephane 
> Bortzmeyer via Outages
> Sent: Tuesday, June 8, 2021 7:10 AM
> To: DaZZa <dazzagibbs at gmail.com>
> Cc: outages at outages.org
> Subject: Re: [outages] Fastly CDN down?
>
> External Email!
>
>
> On Tue, Jun 08, 2021 at 08:54:48PM +1000,  DaZZa <dazzagibbs at gmail.com> wrote  a message of 45 lines which said:
>
> > Yes. Apparently a widespread outage. Numerous sites affected, 
> > including Reddit.
>
> Seems fixed now.
>
> https://status.fastly.com/
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
_______________________________________________
Outages-discussion mailing list
Outages-discussion at outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion