[VoiceOps] Bandwidth - Monday Outage

[Mike Hammett]

" I'll be disappointed if it was a 1Gbps sustained issue that took them down, I sure hope not." 


Well, it depends on the attack type. Is it volumetric, or is it attacking compute resources? 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 



----- Original Message -----

From: "Peter Beckman" <beckman at angryox.com> 
To: "Ryan Delgrosso" <ryandelgrosso at gmail.com> 
Cc: voiceops at voiceops.org 
Sent: Tuesday, September 28, 2021 2:25:30 PM 
Subject: Re: [VoiceOps] Bandwidth - Monday Outage 

On Tue, 28 Sep 2021, Ryan Delgrosso wrote: 

> Yep, except that 
> 
> A: Bandwidth had to know this is a when not an if. In today's internet if 
> your company can be considered critical infra, you will be attacked. The more 
> likley scenario is the technical staff knew this but the MBA types said they 
> were paranoid delusions and denied the project budget. 

They might have planned for a certain scale, but if they are getting with 
with 100s of Gigabits or Terabits of traffic, they probably are not in a 
situation where the cost of having that infrastructure was reasonable. 

Bandwidth likely does not have multiple 10Tb links with multiple carriers. 

> B: I believe they need to be drawing national attention to this to highlight 
> what a steaming dumpster fire much of the critical infra really is. Mostly 
> because its designed to maximize quarterly earnings, not stay working in the 
> face of adversity. 

Until things are attacked, people are willfully ignorant. Proactive Red Team 
attacks on infrastructure is really the best way to find out from someone 
on your side where your infrastructure is vulnerable. But you gotta wanna 
know where your vulnerabilities are and be willing to pay to find them. 
Capitalism beats out rational thought. 

> C: I'm absolutely sympathetic to their plight having been through a crippling 
> DDOS in a past life which spurred the complete redesign of the entire network 
> into sacrificial pods with more robust transport, and a triage runbook to 
> keep the most things available in the face of an insurmountable onslaught. 

Yup. It's hard to find, hire, and keep engaged people who know how to do 
mitigate DDoS attacks at the level that these attacks are occurring. It's 
gotta be multiple Tbps IMHO. I'll be disappointed if it was a 1Gbps 
sustained issue that took them down, I sure hope not. 

> D: Why is the discussion not yet turning to the fact that all major eyeball 
> networks in the US still don't implement BCP38 as a matter of laziness (or 
> above MBA reasons), and this is what allows these attacks to happen. The 
> telco guys are being held to the STIR/SHAKEN standard over robocalling but 
> for decades the major US ISP's could have implemented network policies that 
> would break the chain of DDOS escalation and don't because they cant be 
> bothered to. 

It seems to take huge failures to get companies to change, and for people 
to change. Once the incident passes, fixing it for the future becomes a 
low-priority task again. Urgent vs Important is a real struggle. 

> I once gave a talk on DDOS at a Carrier fraud association task force meeting 
> (cfca.org) and had representatives from every major US eyeball network in the 
> room and asked the above question and the overwhelming answer I got is 
> "leadership doesn't feel its a worthwhile risk/reward to implement". 

Because it's not worth preventing until it hurts financially. 

Maybe the DDoS actors are really just trying to get more companies to 
improve their networks and are just a bunch of white hats forcing 
companies to do better. 

OK, probably not. 

The good news is that BW likely will have some excellent infrastructure 
improvements over the next few weeks/months that will increase my 
confidence in them. Hopefully. This is the first major ongoing issue I've 
seen with BW in 6 years. 

Outages happen. Mistakes made. You either trust your vendor to get it 
right or you leave and hope the new one is better, lacking any trust built 
up that you had. 

--------------------------------------------------------------------------- 
Peter Beckman Internet Guy 
beckman at angryox.com https://www.angryox.com/ 
--------------------------------------------------------------------------- 
_______________________________________________ 
VoiceOps mailing list 
VoiceOps at voiceops.org 
https://puck.nether.net/mailman/listinfo/voiceops 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20210929/aa2a8643/attachment.htm>