[VoiceOps] Bandwidth East Coast Issues

[Peter Beckman]

Looks like things are recovering.

Our monitoring showed recovery around 21:47 to 21:52 UTC, about 30 minutes
ago.

Latest Update

 	Identified - Bandwidth has identified the cause of this incident and is
 	observing improved network and portal behavior.

 	Sep 25, 18:18 EDT

All of my alarms have cleared, SMS messages are coming in reliably, Inbound
and Outbound calling seem restored in our tests.

Beckman

On Sat, 25 Sep 2021, Peter Beckman wrote:

> Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC.
> Monitoring these two IPs
>
> 		67.231.9.112
> 		67.231.13.113
>
> On Sat, 25 Sep 2021, Peter Beckman wrote:
>
>> Another Update
>>
>> 	Update - Bandwidth teams continue to investigate Voice and Messaging
>> 	Failures. 911 calls are experiencing failures.
>>
>> 	Customers may also experience portal and portal API time-outs, 
>> slowness, or
>> 	unresponsiveness.
>>
>> 	All teams are working towards a resolution at this time.
>>
>> 	Sep 25, 16:34 EDT
>> 
>> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm EDT)
>> as they are now responding at an acceptable rate. I just tested outbound
>> and got a call through to a Verizon Mobile, though the PDD was about 5-6
>> seconds.
>> 
>> Inbound seems to be working, albeit a bit slower than expected.
>> 
>> Seems like we're on the path to full recovery.
>> 
>> Beckman
>> 
>> On Sat, 25 Sep 2021, Peter Beckman wrote:
>> 
>>> After a long silence, no updates other than "we're working on it."
>>>
>>> 	Update - Bandwidth teams continue to investigate Voice and Messaging 
>>> Failures.
>>>
>>> 	At this time, 911 calls are not impacted.
>>>
>>> 	Customers may also experience portal and portal API time-outs, 
>>> slowness, or unresponsiveness.
>>>
>>> 	All teams are working towards a resolution at this time.
>>>
>>> 	Sep 25, 16:22 EDT
>>> 
>>> We are still seeing some calls get through. SMS traffic is being delivered
>>> faster and we have not seen any missed SMS messages. Worst SMS delay was 
>>> 9m
>>> 37s. Last 5 tests in the last 50 minutes were all under 21 seconds, 3 of 5
>>> were under 2 seconds.
>>> 
>>> About half of our calls getting through are Canada DIDs.
>>> 
>>> Beckman
>>> 
>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>> 
>>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>>
>>>>    Investigating - Bandwidth is investigating an incident impacting Voice
>>>>    and Messaging Services. Calls and Messages may experience unexpected
>>>>    failures. All teams are actively engaged.
>>>>
>>>> 	Sep 25, 15:31 EDT
>>>> 
>>>> I will update here as I get updates.
>>>> 
>>>> Outbound calls still failing with timeouts, Inbound calls failing, SMS
>>>> messages are delayed, the delay now up to 8 minutes.
>>>> 
>>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>>> 
>>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice and 
>>>>> SMS
>>>>> starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep has 
>>>>> acknowledged
>>>>> issues are detected but their status page as of yet has not been 
>>>>> updated.
>>>>> 
>>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs receive a
>>>>> "call cannot be completed" error on Verizon, and our SMS tests every 10
>>>>> minutes that usually complete in under 2 seconds are taking 15 seconds 
>>>>> to
>>>>> 3.65 minutes to reach us.
>>>>> 
>>>>> Hope they find the culprit and get it fixed quickly!
>>>>> 
>>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>>> 
>>>>> https://status.bandwidth.com/
>>>> _______________________________________________
>>>> VoiceOps mailing list
>>>> VoiceOps at voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>
>>> 
>>> ---------------------------------------------------------------------------
>>> Peter Beckman                                                  Internet 
>>> Guy
>>> beckman at angryox.com 
>>> https://www.angryox.com/
>>> 
>>> ---------------------------------------------------------------------------
>>> 
>> 
>> ---------------------------------------------------------------------------
>> Peter Beckman                                                  Internet Guy
>> beckman at angryox.com                                https://www.angryox.com/
>> ---------------------------------------------------------------------------
>> 
>
> ---------------------------------------------------------------------------
> Peter Beckman                                                  Internet Guy
> beckman at angryox.com                                https://www.angryox.com/
> ---------------------------------------------------------------------------
>

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                https://www.angryox.com/
---------------------------------------------------------------------------