[VoiceOps] Bandwidth East Coast Issues
Peter Beckman
beckman at angryox.com
Sat Sep 25 19:11:50 EDT 2021
Another Update
Update - Bandwidth continues to see portal and network improvements. At
this time, 911 & messaging traffic is processing as expected. We
continue to work to resolve other impacted services.
Sep 25, 19:07 EDT
The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound
and Outbound is still entirely down. Our tests show that Inbound and
Outbound are functioning, as is SMS messaging and the API.
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
> According to BW TAC it was a DDoS.
>
> The outages list posted this:
>
> Bandwidth is currently experiencing a DDoS attack which is
> intermittently impacting our network. Our network operations and
> engineering teams are actively engaged in critical efforts to protect
> our network and fully restore all services as rapidly as possible.
>
> On Sat, 25 Sep 2021, Peter Beckman wrote:
>
>> Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage.
>> 19:00 to 21:47 UTC
>>
>> We haven't completely determined whether or not this was a full or partial
>> outage, as we still saw some calls come in, but termination had issues as
>> did SMS delivery for a portion of the outage period.
>>
>> Latest BW Update
>>
>> Update - Bandwidth continues to see portal and network improvements. At
>> this time, messaging traffic is processing as expected. We continue to
>> work to resolve other impacted services.
>>
>> Sep 25, 18:42 EDT
>>
>>
>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>
>>> Looks like things are recovering.
>>>
>>> Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes
>>> ago.
>>>
>>> Latest Update
>>>
>>> Identified - Bandwidth has identified the cause of this incident and
>>> is
>>> observing improved network and portal behavior.
>>>
>>> Sep 25, 18:18 EDT
>>>
>>> All of my alarms have cleared, SMS messages are coming in reliably,
>>> Inbound
>>> and Outbound calling seem restored in our tests.
>>>
>>> Beckman
>>>
>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>
>>>> Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC.
>>>> Monitoring these two IPs
>>>>
>>>> 67.231.9.112
>>>> 67.231.13.113
>>>>
>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>
>>>>> Another Update
>>>>>
>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>> Failures. 911 calls are experiencing failures.
>>>>>
>>>>> Customers may also experience portal and portal API time-outs,
>>>>> slowness, or
>>>>> unresponsiveness.
>>>>>
>>>>> All teams are working towards a resolution at this time.
>>>>>
>>>>> Sep 25, 16:34 EDT
>>>>>
>>>>> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm
>>>>> EDT)
>>>>> as they are now responding at an acceptable rate. I just tested outbound
>>>>> and got a call through to a Verizon Mobile, though the PDD was about 5-6
>>>>> seconds.
>>>>>
>>>>> Inbound seems to be working, albeit a bit slower than expected.
>>>>>
>>>>> Seems like we're on the path to full recovery.
>>>>>
>>>>> Beckman
>>>>>
>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>
>>>>>> After a long silence, no updates other than "we're working on it."
>>>>>>
>>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>>> Failures.
>>>>>>
>>>>>> At this time, 911 calls are not impacted.
>>>>>>
>>>>>> Customers may also experience portal and portal API time-outs,
>>>>>> slowness, or unresponsiveness.
>>>>>>
>>>>>> All teams are working towards a resolution at this time.
>>>>>>
>>>>>> Sep 25, 16:22 EDT
>>>>>>
>>>>>> We are still seeing some calls get through. SMS traffic is being
>>>>>> delivered
>>>>>> faster and we have not seen any missed SMS messages. Worst SMS delay
>>>>>> was 9m
>>>>>> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3
>>>>>> of 5
>>>>>> were under 2 seconds.
>>>>>>
>>>>>> About half of our calls getting through are Canada DIDs.
>>>>>>
>>>>>> Beckman
>>>>>>
>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>
>>>>>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>>>>>
>>>>>>> Investigating - Bandwidth is investigating an incident impacting
>>>>>>> Voice
>>>>>>> and Messaging Services. Calls and Messages may experience
>>>>>>> unexpected
>>>>>>> failures. All teams are actively engaged.
>>>>>>>
>>>>>>> Sep 25, 15:31 EDT
>>>>>>>
>>>>>>> I will update here as I get updates.
>>>>>>>
>>>>>>> Outbound calls still failing with timeouts, Inbound calls failing, SMS
>>>>>>> messages are delayed, the delay now up to 8 minutes.
>>>>>>>
>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>
>>>>>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice
>>>>>>>> and SMS
>>>>>>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has
>>>>>>>> acknowledged
>>>>>>>> issues are detected but their status page as of yet has not been
>>>>>>>> updated.
>>>>>>>>
>>>>>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs receive
>>>>>>>> a
>>>>>>>> "call cannot be completed" error on Verizon, and our SMS tests every
>>>>>>>> 10
>>>>>>>> minutes that usually complete in under 2 seconds are taking 15
>>>>>>>> seconds to
>>>>>>>> 3.65 minutes to reach us.
>>>>>>>>
>>>>>>>> Hope they find the culprit and get it fixed quickly!
>>>>>>>>
>>>>>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>>>>>>
>>>>>>>> https://status.bandwidth.com/
>>>>>>> _______________________________________________
>>>>>>> VoiceOps mailing list
>>>>>>> VoiceOps at voiceops.org
>>>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>> Peter Beckman Internet
>>>>>> Guy
>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>> Peter Beckman Internet
>>>>> Guy
>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>> Peter Beckman Internet
>>>> Guy
>>>> beckman at angryox.com https://www.angryox.com/
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------------
>>> Peter Beckman Internet
>>> Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>>> ---------------------------------------------------------------------------
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet Guy
>> beckman at angryox.com https://www.angryox.com/
>> ---------------------------------------------------------------------------
>>
>
> ---------------------------------------------------------------------------
> Peter Beckman Internet Guy
> beckman at angryox.com https://www.angryox.com/
> ---------------------------------------------------------------------------
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
More information about the VoiceOps
mailing list