[VoiceOps] Bandwidth - Monday Outage

[Ryan Delgrosso]

Yep, except that

A: Bandwidth had to know this is a when not an if. In today's internet 
if your company can be considered critical infra, you will be attacked. 
The more likley scenario is the technical staff knew this but the MBA 
types said they were paranoid delusions and denied the project budget.

B: I believe they need to be drawing national attention to this to 
highlight what a steaming dumpster fire much of the critical infra 
really is. Mostly because its designed to maximize quarterly earnings, 
not stay working in the face of adversity.

C: I'm absolutely sympathetic to their plight having been through a 
crippling DDOS in a past life which spurred the complete redesign of the 
entire network into sacrificial pods with more robust transport, and a 
triage runbook to keep the most things available in the face of an 
insurmountable onslaught.

D: Why is the discussion not yet turning to the fact that all major 
eyeball networks in the US still don't implement BCP38 as a matter of 
laziness (or above MBA reasons), and this is what allows these attacks 
to happen. The telco guys are being held to the STIR/SHAKEN standard 
over robocalling but for decades the major US ISP's could have 
implemented network policies that would break the chain of DDOS 
escalation and don't because they cant be bothered to.

I once gave a talk on DDOS at a Carrier fraud association task force 
meeting (cfca.org) and had representatives from every major US eyeball 
network in the room and asked the above question and the overwhelming 
answer I got is "leadership doesn't feel its a worthwhile risk/reward to 
implement".

-Ryan

On 9/27/2021 7:17 PM, Peter Beckman wrote:
> On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>
>> Nothing meaningful other than the normal public party line.
>>
>> I too have heard unofficially that its DDOS, which makes sense given 
>> the recurring nature.
>>
>> 4.5hrs down Sat
>
>  Our monitoring showed 2 hours 47 minutes of actual service affecting
>  outages across Voice (Inbound and Outbound), Messaging, and API/Portal.
>
>  The issue started at 3pm and recovered at 5:47pm EDT. We reported it to
>  the TAC at 3:07pm, they did not post on Status until 3:31pm.
>
>> Some small downtime Sun
>>
>> Now deep into Monday with problems.
>>
>> Its not a good look, but id like some more transparency.
>
>  DDoS attacks are real and hard to null route. You've got millions of IP
>  addresses slamming you with data. Your router has a capacity, and your
>  router cannot handle all of that extra crap data along with all of our
>  traffic too.
>
>  I'm sure BW will be investing in some beefy hardware that will be 
> able to
>  better handle DDoS attacks, as well as working more closely with their
>  peering providers. I have to assume that they were getting gigabits of
>  traffic, overwhelming their links in addition to their edge routers.
>
>  Cloudflare details how they do it here:
>  https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection 
>
>
>  Not much to be transparent about. The Internet is an unfriendly 
> place, and
>  bad actors can rain hell upon any public IP they want. Unsecured 
> laptops,
>  desktops, TVs, IOT devices, etc, all contribute just a little tiny bit,
>  and all focus on one single point, kinda like those giant solar farms 
> with
>  the mirrors and single tower in the middle to boil the molten salt.
>
>  Well, Bandwidth is the molten salt, and the mirrors are a bunch of
>  unsecured devices on the Internet.
>
> --------------------------------------------------------------------------- 
>
> Peter Beckman Internet Guy
> beckman at angryox.com https://www.angryox.com/
> ---------------------------------------------------------------------------
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20210928/1d6e4cdb/attachment-0001.htm>