[VoiceOps] Bandwidth East Coast Issues
Peter Beckman
beckman at angryox.com
Sat Sep 25 19:31:42 EDT 2021
BW Status Page is showing 100% operational now. Our monitoring confirms
their supposition.
Monitoring - Bandwidth teams have seen overall network stability with
expected traffic behavior. We will continue to monitor the network.
Sep 25, 19:29 EDT
On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
> Another Update
>
> Update - Bandwidth continues to see portal and network improvements. At
> this time, 911 & messaging traffic is processing as expected. We
> continue to work to resolve other impacted services.
>
> Sep 25, 19:07 EDT
>
> The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound
> and Outbound is still entirely down. Our tests show that Inbound and
> Outbound are functioning, as is SMS messaging and the API.
>
>
> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>
>> According to BW TAC it was a DDoS.
>>
>> The outages list posted this:
>>
>> Bandwidth is currently experiencing a DDoS attack which is
>> intermittently impacting our network. Our network operations and
>> engineering teams are actively engaged in critical efforts to protect
>> our network and fully restore all services as rapidly as possible.
>>
>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>
>>> Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage.
>>> 19:00 to 21:47 UTC
>>>
>>> We haven't completely determined whether or not this was a full or partial
>>> outage, as we still saw some calls come in, but termination had issues as
>>> did SMS delivery for a portion of the outage period.
>>>
>>> Latest BW Update
>>>
>>> Update - Bandwidth continues to see portal and network improvements. At
>>> this time, messaging traffic is processing as expected. We continue to
>>> work to resolve other impacted services.
>>>
>>> Sep 25, 18:42 EDT
>>>
>>>
>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>
>>>> Looks like things are recovering.
>>>>
>>>> Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30
>>>> minutes
>>>> ago.
>>>>
>>>> Latest Update
>>>>
>>>> Identified - Bandwidth has identified the cause of this incident and
>>>> is
>>>> observing improved network and portal behavior.
>>>>
>>>> Sep 25, 18:18 EDT
>>>>
>>>> All of my alarms have cleared, SMS messages are coming in reliably,
>>>> Inbound
>>>> and Outbound calling seem restored in our tests.
>>>>
>>>> Beckman
>>>>
>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>
>>>>> Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38
>>>>> UTC.
>>>>> Monitoring these two IPs
>>>>>
>>>>> 67.231.9.112
>>>>> 67.231.13.113
>>>>>
>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>
>>>>>> Another Update
>>>>>>
>>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>>> Failures. 911 calls are experiencing failures.
>>>>>>
>>>>>> Customers may also experience portal and portal API time-outs,
>>>>>> slowness, or
>>>>>> unresponsiveness.
>>>>>>
>>>>>> All teams are working towards a resolution at this time.
>>>>>>
>>>>>> Sep 25, 16:34 EDT
>>>>>>
>>>>>> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm
>>>>>> EDT)
>>>>>> as they are now responding at an acceptable rate. I just tested
>>>>>> outbound
>>>>>> and got a call through to a Verizon Mobile, though the PDD was about
>>>>>> 5-6
>>>>>> seconds.
>>>>>>
>>>>>> Inbound seems to be working, albeit a bit slower than expected.
>>>>>>
>>>>>> Seems like we're on the path to full recovery.
>>>>>>
>>>>>> Beckman
>>>>>>
>>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>>
>>>>>>> After a long silence, no updates other than "we're working on it."
>>>>>>>
>>>>>>> Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>>>> Failures.
>>>>>>>
>>>>>>> At this time, 911 calls are not impacted.
>>>>>>>
>>>>>>> Customers may also experience portal and portal API time-outs,
>>>>>>> slowness, or unresponsiveness.
>>>>>>>
>>>>>>> All teams are working towards a resolution at this time.
>>>>>>>
>>>>>>> Sep 25, 16:22 EDT
>>>>>>>
>>>>>>> We are still seeing some calls get through. SMS traffic is being
>>>>>>> delivered
>>>>>>> faster and we have not seen any missed SMS messages. Worst SMS delay
>>>>>>> was 9m
>>>>>>> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3
>>>>>>> of 5
>>>>>>> were under 2 seconds.
>>>>>>>
>>>>>>> About half of our calls getting through are Canada DIDs.
>>>>>>>
>>>>>>> Beckman
>>>>>>>
>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>
>>>>>>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>>>>>>
>>>>>>>> Investigating - Bandwidth is investigating an incident impacting
>>>>>>>> Voice
>>>>>>>> and Messaging Services. Calls and Messages may experience
>>>>>>>> unexpected
>>>>>>>> failures. All teams are actively engaged.
>>>>>>>>
>>>>>>>> Sep 25, 15:31 EDT
>>>>>>>>
>>>>>>>> I will update here as I get updates.
>>>>>>>>
>>>>>>>> Outbound calls still failing with timeouts, Inbound calls failing,
>>>>>>>> SMS
>>>>>>>> messages are delayed, the delay now up to 8 minutes.
>>>>>>>>
>>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>>
>>>>>>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice
>>>>>>>>> and SMS
>>>>>>>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has
>>>>>>>>> acknowledged
>>>>>>>>> issues are detected but their status page as of yet has not been
>>>>>>>>> updated.
>>>>>>>>>
>>>>>>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs receive
>>>>>>>>> a
>>>>>>>>> "call cannot be completed" error on Verizon, and our SMS tests every
>>>>>>>>> 10
>>>>>>>>> minutes that usually complete in under 2 seconds are taking 15
>>>>>>>>> seconds to
>>>>>>>>> 3.65 minutes to reach us.
>>>>>>>>>
>>>>>>>>> Hope they find the culprit and get it fixed quickly!
>>>>>>>>>
>>>>>>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>>>>>>>
>>>>>>>>> https://status.bandwidth.com/
>>>>>>>> _______________________________________________
>>>>>>>> VoiceOps mailing list
>>>>>>>> VoiceOps at voiceops.org
>>>>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------------
>>>>>>> Peter Beckman
>>>>>>> Internet Guy
>>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------------
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>> Peter Beckman Internet
>>>>>> Guy
>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------------
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>> Peter Beckman Internet
>>>>> Guy
>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------------
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>> Peter Beckman Internet
>>>> Guy
>>>> beckman at angryox.com https://www.angryox.com/
>>>>
>>>>
>>>> ---------------------------------------------------------------------------
>>>> _______________________________________________
>>>> VoiceOps mailing list
>>>> VoiceOps at voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------------
>>> Peter Beckman Internet
>>> Guy
>>> beckman at angryox.com
>>> https://www.angryox.com/
>>>
>>> ---------------------------------------------------------------------------
>>>
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman Internet Guy
>> beckman at angryox.com https://www.angryox.com/
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>
> ---------------------------------------------------------------------------
> Peter Beckman Internet Guy
> beckman at angryox.com https://www.angryox.com/
> ---------------------------------------------------------------------------
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>
---------------------------------------------------------------------------
Peter Beckman Internet Guy
beckman at angryox.com https://www.angryox.com/
---------------------------------------------------------------------------
More information about the VoiceOps
mailing list