[Outages-discussion] FW: [outages] Second Bandwidth Outage Sunday

[Sean Sailors]

Most of our VOIP customers are complaining as well about the Bandwidth thing.


From: Outages-discussion <outages-discussion-bounces at outages.org> On Behalf Of Eric Herron
Sent: Monday, September 27, 2021 1:05 PM
To: outages-discussion at outages.org
Subject: Re: [Outages-discussion] [outages] Second Bandwidth Outage Sunday

Multiple TACs have confirmed DDoS attacks on their network and they are continuing their mitigation efforts but it appears to be ongoing and intermittent.

My inbound/outbound has been working today but I have had DTMF not work at times throughout the day.

It seems like you may get caught up in it or may not and a subset of services may be affected at any given time.



From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> On Behalf Of Drew Matthews via Outages
Sent: Monday, September 27, 2021 1:54 PM
To: outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] Second Bandwidth Outage Sunday

i believe they are having issues again, not sure if its ddos related. but they are experiencing some major issues. https://status.bandwidth.com/<https://urldefense.proofpoint.com/v2/url?u=https-3A__status.bandwidth.com_&d=DwMGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=BDTo3zD5mrHq76g_7qcoflUzLb6xwPZTKmhMmagZVNU&m=p_8LMWA3SliW7dnfE4HPOaFuo3Vag47Qwy6X4QPbBm4&s=Ue5ZN_IWcpkpEW0gFYkLcB90O1tEYf5NOydjybfz6h8&e=>

Update - Bandwidth teams are also investigating delays with Toll-Free Messaging at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions.
Sep 27, 13:01 EDT
Update - We continue to investigate this incident.
Sep 27, 11:57 EDT
Update - Bandwidth teams are also investigating delayed ports at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions.
Sep 27, 11:07 EDT
Update - We continue to investigate this incidnet.
Sep 27, 11:02 EDT
Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, 911 traffic, Portal & API with intermittent service disruption.
Sep 27, 10:00 EDT
Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, and 911 traffic with intermittent service disruption.
Sep 27, 09:42 EDT
Investigating - Bandwidth teams are investigating an incident impacting our V2 Voice services. Customers may notice intermittent service disruptions.
Sep 27, 09:12 EDT

On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at
11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at
Bandwidth. The impact to us was much reduced this time, as we saw some flapping
of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to
fall below our thresholds.

The API started acting up at 14:56:00 UTC, but recovered, and then has been
flapping between OK and degraded since then. As of 21:52 UTC, the API and
Portal/Dashboard are either non-responsive, slow, or degraded.

Inbound Voice Traffic has remaind at expected historical levels throughout.
I have not checked on Termination, but it was working earlier today without
issue.

Latest from BW https://status.bandwidth.com/<https://urldefense.proofpoint.com/v2/url?u=https-3A__status.bandwidth.com_&d=DwMGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=BDTo3zD5mrHq76g_7qcoflUzLb6xwPZTKmhMmagZVNU&m=p_8LMWA3SliW7dnfE4HPOaFuo3Vag47Qwy6X4QPbBm4&s=Ue5ZN_IWcpkpEW0gFYkLcB90O1tEYf5NOydjybfz6h8&e=>

     Update - Bandwidth teams have observed expected call behavior for 911
     calls. 911 calls are processing as designed and are fully restored. Teams
     continue to work on standard voice traffic.

     Sep 26, 17:20 EDT

I question the phrase "processing as designed" which implies it is not
working as the end user expects... odd.

Beckman
---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com<mailto:beckman at angryox.com>                                https://www.angryox.com/<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.angryox.com_&d=DwMGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=BDTo3zD5mrHq76g_7qcoflUzLb6xwPZTKmhMmagZVNU&m=p_8LMWA3SliW7dnfE4HPOaFuo3Vag47Qwy6X4QPbBm4&s=LujDJI9ijucs-p7m3ncdeqRet5bz1KjUXVV6brFyllk&e=>
---------------------------------------------------------------------------
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages<https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=BDTo3zD5mrHq76g_7qcoflUzLb6xwPZTKmhMmagZVNU&m=p_8LMWA3SliW7dnfE4HPOaFuo3Vag47Qwy6X4QPbBm4&s=Zs9LrL5PqbMvcLyMMgsLuHwYFI29rH4bVLGn7MMGano&e=>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20210927/b0c6d0f7/attachment-0001.htm>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ATT00001.txt
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20210927/b0c6d0f7/attachment-0001.txt>