[VoiceOps] Bandwidth East Coast Issues

Peter Beckman beckman at angryox.com
Sat Sep 25 18:47:39 EDT 2021


Our measurement of the outage was 3pm to 5:47pm US EDT, 2h 47m outage.
     19:00 to 21:47 UTC

We haven't completely determined whether or not this was a full or partial
outage, as we still saw some calls come in, but termination had issues as
did SMS delivery for a portion of the outage period.

Latest BW Update

     Update - Bandwidth continues to see portal and network improvements. At
     this time, messaging traffic is processing as expected. We continue to
     work to resolve other impacted services.

 	Sep 25, 18:42 EDT


On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:

> Looks like things are recovering.
>
> Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes
> ago.
>
> Latest Update
>
> 	Identified - Bandwidth has identified the cause of this incident and 
> is
> 	observing improved network and portal behavior.
>
> 	Sep 25, 18:18 EDT
>
> All of my alarms have cleared, SMS messages are coming in reliably, Inbound
> and Outbound calling seem restored in our tests.
>
> Beckman
>
> On Sat, 25 Sep 2021, Peter Beckman wrote:
>
>> Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC.
>> Monitoring these two IPs
>>
>> 		67.231.9.112
>> 		67.231.13.113
>> 
>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>> 
>>> Another Update
>>>
>>> 	Update - Bandwidth teams continue to investigate Voice and Messaging
>>> 	Failures. 911 calls are experiencing failures.
>>>
>>> 	Customers may also experience portal and portal API time-outs, 
>>> slowness, or
>>> 	unresponsiveness.
>>>
>>> 	All teams are working towards a resolution at this time.
>>>
>>> 	Sep 25, 16:34 EDT
>>> 
>>> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT)
>>> as they are now responding at an acceptable rate. I just tested outbound
>>> and got a call through to a Verizon Mobile, though the PDD was about 5-6
>>> seconds.
>>> 
>>> Inbound seems to be working, albeit a bit slower than expected.
>>> 
>>> Seems like we're on the path to full recovery.
>>> 
>>> Beckman
>>> 
>>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>>> 
>>>> After a long silence, no updates other than "we're working on it."
>>>>
>>>> 	Update - Bandwidth teams continue to investigate Voice and Messaging 
>>>> Failures.
>>>>
>>>> 	At this time, 911 calls are not impacted.
>>>>
>>>> 	Customers may also experience portal and portal API time-outs, 
>>>> slowness, or unresponsiveness.
>>>>
>>>> 	All teams are working towards a resolution at this time.
>>>>
>>>> 	Sep 25, 16:22 EDT
>>>> 
>>>> We are still seeing some calls get through. SMS traffic is being 
>>>> delivered
>>>> faster and we have not seen any missed SMS messages. Worst SMS delay was 
>>>> 9m
>>>> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 
>>>> 5
>>>> were under 2 seconds.
>>>> 
>>>> About half of our calls getting through are Canada DIDs.
>>>> 
>>>> Beckman
>>>> 
>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>> 
>>>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>>>
>>>>>    Investigating - Bandwidth is investigating an incident impacting 
>>>>> Voice
>>>>>    and Messaging Services. Calls and Messages may experience unexpected
>>>>>    failures. All teams are actively engaged.
>>>>>
>>>>> 	Sep 25, 15:31 EDT
>>>>> 
>>>>> I will update here as I get updates.
>>>>> 
>>>>> Outbound calls still failing with timeouts, Inbound calls failing, SMS
>>>>> messages are delayed, the delay now up to 8 minutes.
>>>>> 
>>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>>> 
>>>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice and 
>>>>>> SMS
>>>>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has 
>>>>>> acknowledged
>>>>>> issues are detected but their status page as of yet has not been 
>>>>>> updated.
>>>>>> 
>>>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs receive a
>>>>>> "call cannot be completed" error on Verizon, and our SMS tests every 10
>>>>>> minutes that usually complete in under 2 seconds are taking 15 seconds 
>>>>>> to
>>>>>> 3.65 minutes to reach us.
>>>>>> 
>>>>>> Hope they find the culprit and get it fixed quickly!
>>>>>> 
>>>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>>>> 
>>>>>> https://status.bandwidth.com/
>>>>> _______________________________________________
>>>>> VoiceOps mailing list
>>>>> VoiceOps at voiceops.org
>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>>> 
>>>> 
>>>>
>>>> 
>>>> ---------------------------------------------------------------------------
>>>> Peter Beckman                                                  Internet 
>>>> Guy
>>>> beckman at angryox.com https://www.angryox.com/
>>>>
>>>> 
>>>> ---------------------------------------------------------------------------
>>>> 
>>>
>>> 
>>> ---------------------------------------------------------------------------
>>> Peter Beckman                                                  Internet 
>>> Guy
>>> beckman at angryox.com 
>>> https://www.angryox.com/
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>> 
>> ---------------------------------------------------------------------------
>> Peter Beckman                                                  Internet Guy
>> beckman at angryox.com                                https://www.angryox.com/
>> ---------------------------------------------------------------------------
>> 
>
> ---------------------------------------------------------------------------
> Peter Beckman                                                  Internet Guy
> beckman at angryox.com                                https://www.angryox.com/
> ---------------------------------------------------------------------------
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
>

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                https://www.angryox.com/
---------------------------------------------------------------------------


More information about the VoiceOps mailing list