[VoiceOps] Bandwidth - Monday Outage

[Mark Wiles]

While we all might love to know what they’ve done to TRY to mitigate the issue; it’s reasonable to assume that they’d be fairly quiet about what they’re doing/trying to do.  Right now, I’d rather them keep a low profile and simply get the issue addressed.  You know they’re hemorrhaging customers left-and-right due to port-aways.


From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Ryan Delgrosso
Sent: Wednesday, September 29, 2021 1:52 PM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] Bandwidth - Monday Outage


FYI a pretty weak but publicly referencable acknowledgement of whats going on

https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,owS2cVWZA1WGtGMAEPu5Ti5eAX1FOEqqPpmk_aMkLeDVGUmFu8zbe-bfN7-I3BmpNDZJ3qFWqtTezgSk_R_ZotZ43dLmcgYlB_u6Qh-e-AkGRe0,&typo=1>


On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
They have publicly acknowledge it as a DDoS ( https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.bandwidth.com%2fblog%2fa-message-to-our-customers-and-partners%2f&c=E,1,uAmO5u5c6u8d8fA2aiZUY71pe5rUngX8otVxHtppAMoqMT4mPT6x-kUwGStbW61Br73eiJFUz_ELBDJljCzgYb-3jTJ4oRlE2hKikfXw-w,,&typo=1> ) , but being pretty tight-lipped with specifics on what it is or how they are mitigating.

On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com<mailto:caalvarez at gmail.com>> wrote:
Is this some sort of ransom event against them maybe?  And what are the rest of you telling your customers?  We seem to have only a few specifically complaining, but those are complaining a lot.


On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic <ivan.kovacevic at startelecom.ca<mailto:ivan.kovacevic at startelecom.ca>> wrote:
Happening again.

https://status.bandwidth.com/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidth.com%2f&c=E,1,WolwFQSZ1OSs3rjO6hgO6OvRKpAzNrbIinIqdFrjiYR6iDxcrIaOmjTwQjb8h9dH4srU-RncK8II-R8Nr7Hs6VVXDGoF_4tEQzedk5uxxsq3FSj8yodwABlgng,,&typo=1>



<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>

Ivan Kovacevic
Co-Founder and VP Client Services<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
 <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
 <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps <voiceops at voiceops.org> wrote:<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
On Mon, 27 Sep 2021, Ryan Delgrosso wrote:

> Nothing meaningful other than the normal public party line.
>
> I too have heard unofficially that its DDOS, which makes sense given the
> recurring nature.
>
> 4.5hrs down Sat

  Our monitoring showed 2 hours 47 minutes of actual service affecting
  outages across Voice (Inbound and Outbound), Messaging, and API/Portal.

  The issue started at 3pm and recovered at 5:47pm EDT. We reported it to
  the TAC at 3:07pm, they did not post on Status until 3:31pm.

> Some small downtime Sun
>
> Now deep into Monday with problems.
>
> Its not a good look, but id like some more transparency.

  DDoS attacks are real and hard to null route. You've got millions of IP
  addresses slamming you with data. Your router has a capacity, and your
  router cannot handle all of that extra crap data along with all of our
  traffic too.

  I'm sure BW will be investing in some beefy hardware that will be able to
  better handle DDoS attacks, as well as working more closely with their
  peering providers. I have to assume that they were getting gigabits of
  traffic, overwhelming their links in addition to their edge routers.

  Cloudflare details how they do it here:
  https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection

  Not much to be transparent about. The Internet is an unfriendly place, and
  bad actors can rain hell upon any public IP they want. Unsecured laptops,
  desktops, TVs, IOT devices, etc, all contribute just a little tiny bit,
  and all focus on one single point, kinda like those giant solar farms with
  the mirrors and single tower in the middle to boil the molten salt.

  Well, Bandwidth is the molten salt, and the mirrors are a bunch of
  unsecured devices on the Internet.

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                https://www.angryox.com/
---------------------------------------------------------------------------_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
 <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>

NOTE: This email message and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message.<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>


<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>

_______________________________________________<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>

VoiceOps mailing list<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>

VoiceOps at voiceops.org<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>

https://puck.nether.net/mailman/listinfo/voiceops<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.startelecom.ca%2f&c=E,1,z1xMwqyQSba2tIyKk3epfyt83pf2_1tWCHxSK_gEIhOKhqWf0AI2Pjim0jG0f0GhZfi9CRSrv_uuignvRskhETaKKEng-Jqv74-nf4cdBQ,,&typo=1>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20210929/5b12e027/attachment-0001.htm>