[VoiceOps] Bandwidth East Coast Issues

Peter Beckman beckman at angryox.com
Sat Sep 25 19:47:52 EDT 2021


BW Considers the incident resolved. Permalink to Status Page for the outage:

     https://status.bandwidth.com/incidents/d09cdj4qljc1

Final Update:

     Resolved - With continued network stability, we consider this incident resolved.

     Sep 25, 2021 - 19:45 EDT

On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:

> BW Status Page is showing 100% operational now. Our monitoring confirms
> their supposition.
>
> 	Monitoring - Bandwidth teams have seen overall network stability with
> 	expected traffic behavior. We will continue to monitor the network.
>
> 	Sep 25, 19:29 EDT
>
> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>
>> Another Update
>>
>>    Update - Bandwidth continues to see portal and network improvements. At
>>    this time, 911 & messaging traffic is processing as expected. We
>>    continue to work to resolve other impacted services.
>>
>> 	Sep 25, 19:07 EDT
>> 
>> The Status Page ( https://status.bandwidth.com/ ) is reporting that Inbound
>> and Outbound is still entirely down. Our tests show that Inbound and
>> Outbound are functioning, as is SMS messaging and the API.
>> 
>> 
>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>> 
>>> According to BW TAC it was a DDoS.
>>> 
>>> The outages list posted this:
>>>
>>>    Bandwidth is currently experiencing a DDoS attack which is
>>>    intermittently impacting our network. Our network operations and
>>>    engineering teams are actively engaged in critical efforts to protect
>>>    our network and fully restore all services as rapidly as possible.
>>> 
>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>> 
>>>> Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage.
>>>>    19:00 to 21:47 UTC
>>>> 
>>>> We haven't completely determined whether or not this was a full or 
>>>> partial
>>>> outage, as we still saw some calls come in, but termination had issues as
>>>> did SMS delivery for a portion of the outage period.
>>>> 
>>>> Latest BW Update
>>>>
>>>>    Update - Bandwidth continues to see portal and network improvements. 
>>>> At
>>>>    this time, messaging traffic is processing as expected. We continue to
>>>>    work to resolve other impacted services.
>>>>
>>>> 	Sep 25, 18:42 EDT
>>>> 
>>>> 
>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>> 
>>>>> Looks like things are recovering.
>>>>> 
>>>>> Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 
>>>>> minutes
>>>>> ago.
>>>>> 
>>>>> Latest Update
>>>>>
>>>>> 	Identified - Bandwidth has identified the cause of this incident and 
>>>>> is
>>>>> 	observing improved network and portal behavior.
>>>>>
>>>>> 	Sep 25, 18:18 EDT
>>>>> 
>>>>> All of my alarms have cleared, SMS messages are coming in reliably, 
>>>>> Inbound
>>>>> and Outbound calling seem restored in our tests.
>>>>> 
>>>>> Beckman
>>>>> 
>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>> 
>>>>>> Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 
>>>>>> UTC.
>>>>>> Monitoring these two IPs
>>>>>>
>>>>>> 		67.231.9.112
>>>>>> 		67.231.13.113
>>>>>> 
>>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>> 
>>>>>>> Another Update
>>>>>>>
>>>>>>> 	Update - Bandwidth teams continue to investigate Voice and Messaging
>>>>>>> 	Failures. 911 calls are experiencing failures.
>>>>>>>
>>>>>>> 	Customers may also experience portal and portal API time-outs, 
>>>>>>> slowness, or
>>>>>>> 	unresponsiveness.
>>>>>>>
>>>>>>> 	All teams are working towards a resolution at this time.
>>>>>>>
>>>>>>> 	Sep 25, 16:34 EDT
>>>>>>> 
>>>>>>> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm 
>>>>>>> EDT)
>>>>>>> as they are now responding at an acceptable rate. I just tested 
>>>>>>> outbound
>>>>>>> and got a call through to a Verizon Mobile, though the PDD was about 
>>>>>>> 5-6
>>>>>>> seconds.
>>>>>>> 
>>>>>>> Inbound seems to be working, albeit a bit slower than expected.
>>>>>>> 
>>>>>>> Seems like we're on the path to full recovery.
>>>>>>> 
>>>>>>> Beckman
>>>>>>> 
>>>>>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>>>>>> 
>>>>>>>> After a long silence, no updates other than "we're working on it."
>>>>>>>>
>>>>>>>> 	Update - Bandwidth teams continue to investigate Voice and 
>>>>>>>> Messaging Failures.
>>>>>>>>
>>>>>>>> 	At this time, 911 calls are not impacted.
>>>>>>>>
>>>>>>>> 	Customers may also experience portal and portal API 
>>>>>>>> time-outs, slowness, or unresponsiveness.
>>>>>>>>
>>>>>>>> 	All teams are working towards a resolution at this time.
>>>>>>>>
>>>>>>>> 	Sep 25, 16:22 EDT
>>>>>>>> 
>>>>>>>> We are still seeing some calls get through. SMS traffic is being 
>>>>>>>> delivered
>>>>>>>> faster and we have not seen any missed SMS messages. Worst SMS delay 
>>>>>>>> was 9m
>>>>>>>> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 
>>>>>>>> of 5
>>>>>>>> were under 2 seconds.
>>>>>>>> 
>>>>>>>> About half of our calls getting through are Canada DIDs.
>>>>>>>> 
>>>>>>>> Beckman
>>>>>>>> 
>>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>> 
>>>>>>>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>>>>>>>
>>>>>>>>>    Investigating - Bandwidth is investigating an incident impacting 
>>>>>>>>> Voice
>>>>>>>>>    and Messaging Services. Calls and Messages may experience 
>>>>>>>>> unexpected
>>>>>>>>>    failures. All teams are actively engaged.
>>>>>>>>>
>>>>>>>>> 	Sep 25, 15:31 EDT
>>>>>>>>> 
>>>>>>>>> I will update here as I get updates.
>>>>>>>>> 
>>>>>>>>> Outbound calls still failing with timeouts, Inbound calls failing, 
>>>>>>>>> SMS
>>>>>>>>> messages are delayed, the delay now up to 8 minutes.
>>>>>>>>> 
>>>>>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>>>>>> 
>>>>>>>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice 
>>>>>>>>>> and SMS
>>>>>>>>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has 
>>>>>>>>>> acknowledged
>>>>>>>>>> issues are detected but their status page as of yet has not been 
>>>>>>>>>> updated.
>>>>>>>>>> 
>>>>>>>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs 
>>>>>>>>>> receive a
>>>>>>>>>> "call cannot be completed" error on Verizon, and our SMS tests 
>>>>>>>>>> every 10
>>>>>>>>>> minutes that usually complete in under 2 seconds are taking 15 
>>>>>>>>>> seconds to
>>>>>>>>>> 3.65 minutes to reach us.
>>>>>>>>>> 
>>>>>>>>>> Hope they find the culprit and get it fixed quickly!
>>>>>>>>>> 
>>>>>>>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>>>>>>>> 
>>>>>>>>>> https://status.bandwidth.com/
>>>>>>>>> _______________________________________________
>>>>>>>>> VoiceOps mailing list
>>>>>>>>> VoiceOps at voiceops.org
>>>>>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>
>>>>>>>> 
>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>> Peter Beckman Internet Guy
>>>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>
>>>>>>>> 
>>>>>>>> ---------------------------------------------------------------------------
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>
>>>>>>> 
>>>>>>> ---------------------------------------------------------------------------
>>>>>>> Peter Beckman 
>>>>>>> Internet Guy
>>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>
>>>>>>> 
>>>>>>> ---------------------------------------------------------------------------
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>
>>>>>> 
>>>>>> ---------------------------------------------------------------------------
>>>>>> Peter Beckman                                                  Internet 
>>>>>> Guy
>>>>>> beckman at angryox.com https://www.angryox.com/
>>>>>> 
>>>>>> 
>>>>>>
>>>>>> 
>>>>>> ---------------------------------------------------------------------------
>>>>>> 
>>>>> 
>>>>> 
>>>>>
>>>>> 
>>>>> ---------------------------------------------------------------------------
>>>>> Peter Beckman                                                  Internet 
>>>>> Guy
>>>>> beckman at angryox.com https://www.angryox.com/
>>>>> 
>>>>>
>>>>> 
>>>>> ---------------------------------------------------------------------------
>>>>> _______________________________________________
>>>>> VoiceOps mailing list
>>>>> VoiceOps at voiceops.org
>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>> 
>>>> 
>>>>
>>>> 
>>>> ---------------------------------------------------------------------------
>>>> Peter Beckman                                                  Internet 
>>>> Guy
>>>> beckman at angryox.com https://www.angryox.com/
>>>>
>>>> 
>>>> ---------------------------------------------------------------------------
>>>> 
>>>
>>> 
>>> ---------------------------------------------------------------------------
>>> Peter Beckman                                                  Internet 
>>> Guy
>>> beckman at angryox.com 
>>> https://www.angryox.com/
>>> 
>>> ---------------------------------------------------------------------------
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> 
>> 
>> ---------------------------------------------------------------------------
>> Peter Beckman                                                  Internet Guy
>> beckman at angryox.com                                https://www.angryox.com/
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>> 
>
> ---------------------------------------------------------------------------
> Peter Beckman                                                  Internet Guy
> beckman at angryox.com                                https://www.angryox.com/
> ---------------------------------------------------------------------------
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                https://www.angryox.com/
---------------------------------------------------------------------------


More information about the VoiceOps mailing list