<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>Yep, except that <br>
</p>
<p>A: Bandwidth had to know this is a when not an if. In today's
internet if your company can be considered critical infra, you
will be attacked. The more likley scenario is the technical staff
knew this but the MBA types said they were paranoid delusions and
denied the project budget. <br>
</p>
<p>B: I believe they need to be drawing national attention to this
to highlight what a steaming dumpster fire much of the critical
infra really is. Mostly because its designed to maximize quarterly
earnings, not stay working in the face of adversity. <br>
</p>
<p>C: I'm absolutely sympathetic to their plight having been through
a crippling DDOS in a past life which spurred the complete
redesign of the entire network into sacrificial pods with more
robust transport, and a triage runbook to keep the most things
available in the face of an insurmountable onslaught. <br>
</p>
<p>D: Why is the discussion not yet turning to the fact that all
major eyeball networks in the US still don't implement BCP38 as a
matter of laziness (or above MBA reasons), and this is what allows
these attacks to happen. The telco guys are being held to the
STIR/SHAKEN standard over robocalling but for decades the major US
ISP's could have implemented network policies that would break the
chain of DDOS escalation and don't because they cant be bothered
to. <br>
</p>
<p>I once gave a talk on DDOS at a Carrier fraud association task
force meeting (cfca.org) and had representatives from every major
US eyeball network in the room and asked the above question and
the overwhelming answer I got is "leadership doesn't feel its a
worthwhile risk/reward to implement". <br>
</p>
<p>-Ryan <br>
</p>
<div class="moz-cite-prefix">On 9/27/2021 7:17 PM, Peter Beckman
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:alpine.BSF.2.20.2109272207030.48288@nog2.angryox.com">On
Mon, 27 Sep 2021, Ryan Delgrosso wrote:
<br>
<br>
<blockquote type="cite">Nothing meaningful other than the normal
public party line.
<br>
<br>
I too have heard unofficially that its DDOS, which makes sense
given the recurring nature.
<br>
<br>
4.5hrs down Sat
<br>
</blockquote>
<br>
Our monitoring showed 2 hours 47 minutes of actual service
affecting
<br>
outages across Voice (Inbound and Outbound), Messaging, and
API/Portal.
<br>
<br>
The issue started at 3pm and recovered at 5:47pm EDT. We reported
it to
<br>
the TAC at 3:07pm, they did not post on Status until 3:31pm.
<br>
<br>
<blockquote type="cite">Some small downtime Sun
<br>
<br>
Now deep into Monday with problems.
<br>
<br>
Its not a good look, but id like some more transparency.
<br>
</blockquote>
<br>
DDoS attacks are real and hard to null route. You've got millions
of IP
<br>
addresses slamming you with data. Your router has a capacity, and
your
<br>
router cannot handle all of that extra crap data along with all
of our
<br>
traffic too.
<br>
<br>
I'm sure BW will be investing in some beefy hardware that will be
able to
<br>
better handle DDoS attacks, as well as working more closely with
their
<br>
peering providers. I have to assume that they were getting
gigabits of
<br>
traffic, overwhelming their links in addition to their edge
routers.
<br>
<br>
Cloudflare details how they do it here:
<br>
<a class="moz-txt-link-freetext" href="https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection">https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection</a>
<br>
<br>
Not much to be transparent about. The Internet is an unfriendly
place, and
<br>
bad actors can rain hell upon any public IP they want. Unsecured
laptops,
<br>
desktops, TVs, IOT devices, etc, all contribute just a little
tiny bit,
<br>
and all focus on one single point, kinda like those giant solar
farms with
<br>
the mirrors and single tower in the middle to boil the molten
salt.
<br>
<br>
Well, Bandwidth is the molten salt, and the mirrors are a bunch
of
<br>
unsecured devices on the Internet.
<br>
<br>
---------------------------------------------------------------------------
<br>
Peter Beckman
Internet Guy
<br>
<a class="moz-txt-link-abbreviated" href="mailto:beckman@angryox.com">beckman@angryox.com</a>
<a class="moz-txt-link-freetext" href="https://www.angryox.com/">https://www.angryox.com/</a>
<br>
---------------------------------------------------------------------------<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
VoiceOps mailing list
<a class="moz-txt-link-abbreviated" href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a>
<a class="moz-txt-link-freetext" href="https://puck.nether.net/mailman/listinfo/voiceops">https://puck.nether.net/mailman/listinfo/voiceops</a>
</pre>
</blockquote>
</body>
</html>