[VoiceOps] Bandwidth East Coast Issues

[Peter Beckman]

Another Update

 	Update - Bandwidth teams continue to investigate Voice and Messaging
 	Failures. 911 calls are experiencing failures.

 	Customers may also experience portal and portal API time-outs, slowness, or
 	unresponsiveness.

 	All teams are working towards a resolution at this time.

 	Sep 25, 16:34 EDT

Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT)
as they are now responding at an acceptable rate. I just tested outbound
and got a call through to a Verizon Mobile, though the PDD was about 5-6
seconds.

Inbound seems to be working, albeit a bit slower than expected.

Seems like we're on the path to full recovery.

Beckman

On Sat, 25 Sep 2021, Peter Beckman wrote:

> After a long silence, no updates other than "we're working on it."
>
> 	Update - Bandwidth teams continue to investigate Voice and Messaging 
> Failures.
>
> 	At this time, 911 calls are not impacted.
>
> 	Customers may also experience portal and portal API time-outs, 
> slowness, or unresponsiveness.
>
> 	All teams are working towards a resolution at this time.
>
> 	Sep 25, 16:22 EDT
>
> We are still seeing some calls get through. SMS traffic is being delivered
> faster and we have not seen any missed SMS messages. Worst SMS delay was 9m
> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5
> were under 2 seconds.
>
> About half of our calls getting through are Canada DIDs.
>
> Beckman
>
> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>
>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>
>>    Investigating - Bandwidth is investigating an incident impacting Voice
>>    and Messaging Services. Calls and Messages may experience unexpected
>>    failures. All teams are actively engaged.
>>
>> 	Sep 25, 15:31 EDT
>> 
>> I will update here as I get updates.
>> 
>> Outbound calls still failing with timeouts, Inbound calls failing, SMS
>> messages are delayed, the delay now up to 8 minutes.
>> 
>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>> 
>>> We began detecting issues with Bandwidth Inbound and Outbound Voice and 
>>> SMS
>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has acknowledged
>>> issues are detected but their status page as of yet has not been updated.
>>> 
>>> I'm seeing outbound calls time out, inbound calls to BW DIDs receive a
>>> "call cannot be completed" error on Verizon, and our SMS tests every 10
>>> minutes that usually complete in under 2 seconds are taking 15 seconds to
>>> 3.65 minutes to reach us.
>>> 
>>> Hope they find the culprit and get it fixed quickly!
>>> 
>>> As of 3:27:45pm EDT their status page shows zero incidents
>>> 
>>> https://status.bandwidth.com/
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
>> 
>
> ---------------------------------------------------------------------------
> Peter Beckman                                                  Internet Guy
> beckman at angryox.com                                https://www.angryox.com/
> ---------------------------------------------------------------------------
>

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                https://www.angryox.com/
---------------------------------------------------------------------------