[VoiceOps] Bandwidth - Monday Outage

[Ryan Delgrosso]

FYI a pretty weak but publicly referencable acknowledgement of whats 
going on

https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/


On 9/29/2021 10:37 AM, Pete Eisengrein wrote:
> They have publicly acknowledge it as a DDoS ( 
> https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/ 
> <https://www.bandwidth.com/blog/a-message-to-our-customers-and-partners/> 
> ) , but being pretty tight-lipped with specifics on what it is or how 
> they are mitigating.
>
> On Wed, Sep 29, 2021 at 12:29 PM Carlos Alvarez <caalvarez at gmail.com 
> <mailto:caalvarez at gmail.com>> wrote:
>
>     Is this some sort of ransom event against them maybe?  And what
>     are the rest of you telling your customers?  We seem to have only
>     a few specifically complaining, but those are complaining a lot.
>
>
>     On Tue, Sep 28, 2021 at 11:06 PM Ivan Kovacevic
>     <ivan.kovacevic at startelecom.ca
>     <mailto:ivan.kovacevic at startelecom.ca>> wrote:
>
>         Happening again.
>
>         https://status.bandwidth.com/ <https://status.bandwidth.com/>
>
>
>         Star Telecom - Cloud Communications and Customer Experience
>         Solutions <https://www.startelecom.ca/>
>
>         	
>         	
>
>         *Ivan Kovacevic*
>         /Co-Founder and VP Client Services
>         /
>
>
>
>
>         On Mon, Sep 27, 2021 at 10:19 PM Peter Beckman via VoiceOps
>         <voiceops at voiceops.org <mailto:voiceops at voiceops.org>> wrote:
>
>             On Mon, 27 Sep 2021, Ryan Delgrosso wrote:
>
>             > Nothing meaningful other than the normal public party line.
>             >
>             > I too have heard unofficially that its DDOS, which makes
>             sense given the
>             > recurring nature.
>             >
>             > 4.5hrs down Sat
>
>               Our monitoring showed 2 hours 47 minutes of actual
>             service affecting
>               outages across Voice (Inbound and Outbound), Messaging,
>             and API/Portal.
>
>               The issue started at 3pm and recovered at 5:47pm EDT. We
>             reported it to
>               the TAC at 3:07pm, they did not post on Status until 3:31pm.
>
>             > Some small downtime Sun
>             >
>             > Now deep into Monday with problems.
>             >
>             > Its not a good look, but id like some more transparency.
>
>               DDoS attacks are real and hard to null route. You've got
>             millions of IP
>               addresses slamming you with data. Your router has a
>             capacity, and your
>               router cannot handle all of that extra crap data along
>             with all of our
>               traffic too.
>
>               I'm sure BW will be investing in some beefy hardware
>             that will be able to
>               better handle DDoS attacks, as well as working more
>             closely with their
>               peering providers. I have to assume that they were
>             getting gigabits of
>               traffic, overwhelming their links in addition to their
>             edge routers.
>
>               Cloudflare details how they do it here:
>             https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection
>             <https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection>
>
>               Not much to be transparent about. The Internet is an
>             unfriendly place, and
>               bad actors can rain hell upon any public IP they want.
>             Unsecured laptops,
>               desktops, TVs, IOT devices, etc, all contribute just a
>             little tiny bit,
>               and all focus on one single point, kinda like those
>             giant solar farms with
>               the mirrors and single tower in the middle to boil the
>             molten salt.
>
>               Well, Bandwidth is the molten salt, and the mirrors are
>             a bunch of
>               unsecured devices on the Internet.
>
>             ---------------------------------------------------------------------------
>             Peter Beckman           Internet Guy
>             beckman at angryox.com <mailto:beckman at angryox.com>
>             https://www.angryox.com/ <https://www.angryox.com/>
>             ---------------------------------------------------------------------------_______________________________________________
>             VoiceOps mailing list
>             VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
>             https://puck.nether.net/mailman/listinfo/voiceops
>             <https://puck.nether.net/mailman/listinfo/voiceops>
>             _______________________________________________
>             VoiceOps mailing list
>             VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
>             https://puck.nether.net/mailman/listinfo/voiceops
>             <https://puck.nether.net/mailman/listinfo/voiceops>
>
>
>         NOTE: This email message and any attachments are for the sole
>         use of the intended recipient(s) and may contain confidential
>         and/or privileged information. Any unauthorized review, use,
>         disclosure or distribution is prohibited. If you are not the
>         intended recipient, please contact the sender by replying to
>         this email, and destroy all copies of the original message.
>
>         _______________________________________________
>         VoiceOps mailing list
>         VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
>         https://puck.nether.net/mailman/listinfo/voiceops
>         <https://puck.nether.net/mailman/listinfo/voiceops>
>
>     _______________________________________________
>     VoiceOps mailing list
>     VoiceOps at voiceops.org <mailto:VoiceOps at voiceops.org>
>     https://puck.nether.net/mailman/listinfo/voiceops
>     <https://puck.nether.net/mailman/listinfo/voiceops>
>
>
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20210929/54a311a6/attachment-0001.htm>