[Outages-discussion] FW: [outages-discussion] Fastly CDN down?

[Jeremy Chadwick]

https://news.ycombinator.com/item?id=27432408 has some discussion about
why certain parts of Amazon (not AWS!) uses Fastly, and what Amazon did
to mitigate (switched to Cloudfront).

As for "who knew half the internet used this same CDN", I've known this
for a while.  The number of companies doing this continue to grow.  For
some reason online service providers have still not learned to avoid
putting all their eggs in one basket; they will spend exorbitant amounts
of time and money to guarantee 5-9s but then introduce critical SPoFs
*by choice*.  Another recent example of this same mindset was the AWS
Kinesis outage in November, where AWS simply trusted their own tech too
much: https://aws.amazon.com/message/11201/

{off-topic}  There is a pervasive mentality in the online web-based
service industry that, put roughly, amounts to "Big Company X uses
Thingus, so we should use Thingus too".  This applies to not just CDNs
or other service providers, but software as well.  Us old codgers know
this isn't a good idea (or if it's deemed acceptable, that the downsides
are well-understood and other mitigation methods are available at time
of infrastructure design).  "Why do it ourselves when Derpaherpaderp.io
can do it for us?" is a dangerous question, and one that should be
approached with extreme caution.  Proper monitoring is almost always
neglected as well (Ops are told there are more important things to
address, rinse lather repeat for years -- a backlog that never gets
worked on).  {/off-topic}

That said, ~1-1.5 hours from start to fix is pretty good:
https://status.fastly.com/#past-incidents .  Respects to those in Fastly
Operations who likely got woken up + had to deal with it.

-- 
| Jeremy Chadwick                                 jdc at koitsu.org |
| UNIX Systems Administrator                      PGP 0x2A389531 |
| Making life hard for others since 1977.                        |

On Tue, Jun 08, 2021 at 11:36:24AM +0000, Mike Castoro wrote:
> Added to Outages-Discussion.  Who knew that half the internet used the same CDN edge provider?   Why would AWS and GCP use them?  Our Shopify sites were down this morning and Shopify support scrambled to even know it was happening (plus their status page was down as well).
> 
> Mike Castoro | VP of IT
> Adrianna Papell Group | 646-723-3885
> Chat with me in Teams
> 
> -----Original Message-----
> From: Outages <outages-bounces at outages.org> On Behalf Of Stephane Bortzmeyer via Outages
> Sent: Tuesday, June 8, 2021 7:10 AM
> To: DaZZa <dazzagibbs at gmail.com>
> Cc: outages at outages.org
> Subject: Re: [outages] Fastly CDN down?
> 
> External Email!
> 
> 
> On Tue, Jun 08, 2021 at 08:54:48PM +1000,  DaZZa <dazzagibbs at gmail.com> wrote  a message of 45 lines which said:
> 
> > Yes. Apparently a widespread outage. Numerous sites affected, 
> > including Reddit.
> 
> Seems fixed now.
> 
> https://status.fastly.com/
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>