<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000'><font face="arial, helvetica, sans-serif"><span style="font-size: 10pt;">"</span><span style="font-size: 13.3333px;">I'll be disappointed if it was a 1Gbps </span></font><span style="font-size: 13.3333px; font-family: arial, helvetica, sans-serif;">sustained issue that took them down, I sure hope not."</span><div><font face="arial, helvetica, sans-serif"><span style="font-size: 13.3333px;"><br></span></font></div><div><font face="arial, helvetica, sans-serif"><span style="font-size: 13.3333px;">Well, it depends on the attack type. Is it volumetric, or is it attacking compute resources?<br></span></font><br><div style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><span name="x"></span><br><br>-----<br>Mike Hammett<br>Intelligent Computing Solutions<br>http://www.ics-il.com<br><br><br><br>Midwest Internet Exchange<br>http://www.midwest-ix.com<br><br><span name="x"></span><br></div><br><hr id="zwchr" style="color: rgb(0, 0, 0); font-family: arial, helvetica, sans-serif; font-size: 10pt;"><div style="color: rgb(0, 0, 0); font-family: Helvetica, Arial, sans-serif; font-size: 12pt; font-weight: normal; font-style: normal; text-decoration: none;"><b>From: </b>"Peter Beckman" <beckman@angryox.com><br><b>To: </b>"Ryan Delgrosso" <ryandelgrosso@gmail.com><br><b>Cc: </b>voiceops@voiceops.org<br><b>Sent: </b>Tuesday, September 28, 2021 2:25:30 PM<br><b>Subject: </b>Re: [VoiceOps] Bandwidth - Monday Outage<br><br>On Tue, 28 Sep 2021, Ryan Delgrosso wrote:<br><br>> Yep, except that<br>><br>> A: Bandwidth had to know this is a when not an if. In today's internet if <br>> your company can be considered critical infra, you will be attacked. The more <br>> likley scenario is the technical staff knew this but the MBA types said they <br>> were paranoid delusions and denied the project budget.<br><br> They might have planned for a certain scale, but if they are getting with<br> with 100s of Gigabits or Terabits of traffic, they probably are not in a<br> situation where the cost of having that infrastructure was reasonable.<br><br> Bandwidth likely does not have multiple 10Tb links with multiple carriers.<br><br>> B: I believe they need to be drawing national attention to this to highlight <br>> what a steaming dumpster fire much of the critical infra really is. Mostly <br>> because its designed to maximize quarterly earnings, not stay working in the <br>> face of adversity.<br><br> Until things are attacked, people are willfully ignorant. Proactive Red Team<br> attacks on infrastructure is really the best way to find out from someone<br> on your side where your infrastructure is vulnerable. But you gotta wanna<br> know where your vulnerabilities are and be willing to pay to find them.<br> Capitalism beats out rational thought.<br><br>> C: I'm absolutely sympathetic to their plight having been through a crippling <br>> DDOS in a past life which spurred the complete redesign of the entire network <br>> into sacrificial pods with more robust transport, and a triage runbook to <br>> keep the most things available in the face of an insurmountable onslaught.<br><br> Yup. It's hard to find, hire, and keep engaged people who know how to do<br> mitigate DDoS attacks at the level that these attacks are occurring. It's<br> gotta be multiple Tbps IMHO. I'll be disappointed if it was a 1Gbps<br> sustained issue that took them down, I sure hope not.<br><br>> D: Why is the discussion not yet turning to the fact that all major eyeball <br>> networks in the US still don't implement BCP38 as a matter of laziness (or <br>> above MBA reasons), and this is what allows these attacks to happen. The <br>> telco guys are being held to the STIR/SHAKEN standard over robocalling but <br>> for decades the major US ISP's could have implemented network policies that <br>> would break the chain of DDOS escalation and don't because they cant be <br>> bothered to.<br><br> It seems to take huge failures to get companies to change, and for people<br> to change. Once the incident passes, fixing it for the future becomes a<br> low-priority task again. Urgent vs Important is a real struggle.<br><br>> I once gave a talk on DDOS at a Carrier fraud association task force meeting <br>> (cfca.org) and had representatives from every major US eyeball network in the <br>> room and asked the above question and the overwhelming answer I got is <br>> "leadership doesn't feel its a worthwhile risk/reward to implement".<br><br> Because it's not worth preventing until it hurts financially.<br><br> Maybe the DDoS actors are really just trying to get more companies to<br> improve their networks and are just a bunch of white hats forcing<br> companies to do better.<br><br> OK, probably not.<br><br> The good news is that BW likely will have some excellent infrastructure<br> improvements over the next few weeks/months that will increase my<br> confidence in them. Hopefully. This is the first major ongoing issue I've<br> seen with BW in 6 years.<br><br> Outages happen. Mistakes made. You either trust your vendor to get it<br> right or you leave and hope the new one is better, lacking any trust built<br> up that you had.<br><br>---------------------------------------------------------------------------<br>Peter Beckman Internet Guy<br>beckman@angryox.com https://www.angryox.com/<br>---------------------------------------------------------------------------<br>_______________________________________________<br>VoiceOps mailing list<br>VoiceOps@voiceops.org<br>https://puck.nether.net/mailman/listinfo/voiceops<br></div><br></div></div></body></html>