[Outages-discussion] [outages] Discord down (fwd)

[Peter Beckman]

Building services with single point-of-failure vendors is probably bad.

For telcos, it is difficult, since if your ratecenter/CLEC/ILEC is down,
there is no alternative, AFAIK.

However, for web services, one should never depend on a single region, and
cloud service providers need to ensure isolation so that a failure in one
region does not take down another region.

Always plan on your primary region to die a horrible death, and have a
mitigation plan to fail to a secondary or tertiary region that is ready and
built and tested. Unless one enjoys downtime... :-)

GCTD seems to imply that this affected only us-east1 repeatedly: "Thus far,
impact has only been observed in us-east1, but customers with
multi-regional Traffic Director deployments could be impacted in more
regions."

Google Cloud Traffic Director Outage:
https://status.cloud.google.com/incidents/LuGcJVjNTeC5Sb9pSJ9o

>From Discord:

Mar 8, 13:01 PST - Resolved - Service is working as expected at this time.

Engineering has root-caused this issue to a Google Cloud component called
"Traffic Director" which is responsible for configuring our load balancing
layer. In its malfunction, it caused our internal load balancing layer to
not have a valid configuration, which caused a loss of availability of the
API. Engineering took measures to remediate by moving to alternate proxy
configuration that did not use traffic director. It took us a bit to switch
to using a different load balancing topology, but we were able to do so to
restore service before Traffic Director issues were resolved.


---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                https://www.angryox.com/
---------------------------------------------------------------------------

---------- Forwarded message ----------
Date: Tue, 8 Mar 2022 14:01:28
From: Cristian Beckerle | Bee via Outages <outages at outages.org>
Reply-To: Cristian Beckerle | Bee <cristian at beewh.com>
To: OUTAGES Mailing List <outages at outages.org>
Subject: [outages] Discord down

Discord is down since more than one hour ago.

https://discordstatus.com/