[VoiceOps] Bandwidth East Coast Issues

[Mark Wiles]

Hmmm... DoS?

-----Original Message-----
From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Peter Beckman via VoiceOps
Sent: Saturday, September 25, 2021 4:49 PM
To: VoiceOps <voiceops at voiceops.org>
Subject: Re: [VoiceOps] Bandwidth East Coast Issues

Spoke too soon, Inbound SBCs are now timing out again as of 20:45:38 UTC.
Monitoring these two IPs

 		67.231.9.112
 		67.231.13.113

On Sat, 25 Sep 2021, Peter Beckman wrote:

> Another Update
>
> 	Update - Bandwidth teams continue to investigate Voice and Messaging
> 	Failures. 911 calls are experiencing failures.
>
> 	Customers may also experience portal and portal API time-outs, 
> slowness, or
> 	unresponsiveness.
>
> 	All teams are working towards a resolution at this time.
>
> 	Sep 25, 16:34 EDT
>
> Our monitoring cleared our Inbound SBCs as of 20:34:24 UTC (4:34:24pm 
> EDT) as they are now responding at an acceptable rate. I just tested 
> outbound and got a call through to a Verizon Mobile, though the PDD 
> was about 5-6 seconds.
>
> Inbound seems to be working, albeit a bit slower than expected.
>
> Seems like we're on the path to full recovery.
>
> Beckman
>
> On Sat, 25 Sep 2021, Peter Beckman wrote:
>
>> After a long silence, no updates other than "we're working on it."
>>
>> 	Update - Bandwidth teams continue to investigate Voice and Messaging 
>> Failures.
>>
>> 	At this time, 911 calls are not impacted.
>>
>> 	Customers may also experience portal and portal API time-outs, 
>> slowness, or unresponsiveness.
>>
>> 	All teams are working towards a resolution at this time.
>>
>> 	Sep 25, 16:22 EDT
>> 
>> We are still seeing some calls get through. SMS traffic is being 
>> delivered faster and we have not seen any missed SMS messages. Worst 
>> SMS delay was 9m 37s. Last 5 tests in the last 50 minutes were all 
>> under 21 seconds, 3 of 5 were under 2 seconds.
>> 
>> About half of our calls getting through are Canada DIDs.
>> 
>> Beckman
>> 
>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>> 
>>> As of 3:31pm EDT Bandwidth has acknowledged the issue:
>>>
>>>    Investigating - Bandwidth is investigating an incident impacting Voice
>>>    and Messaging Services. Calls and Messages may experience unexpected
>>>    failures. All teams are actively engaged.
>>>
>>> 	Sep 25, 15:31 EDT
>>> 
>>> I will update here as I get updates.
>>> 
>>> Outbound calls still failing with timeouts, Inbound calls failing, 
>>> SMS messages are delayed, the delay now up to 8 minutes.
>>> 
>>> On Sat, 25 Sep 2021, Peter Beckman via VoiceOps wrote:
>>> 
>>>> We began detecting issues with Bandwidth Inbound and Outbound Voice 
>>>> and SMS starting at 19:00:38 UTC (3pm US Eastern Time). A TAC rep 
>>>> has acknowledged issues are detected but their status page as of 
>>>> yet has not been updated.
>>>> 
>>>> I'm seeing outbound calls time out, inbound calls to BW DIDs 
>>>> receive a "call cannot be completed" error on Verizon, and our SMS 
>>>> tests every 10 minutes that usually complete in under 2 seconds are 
>>>> taking 15 seconds to
>>>> 3.65 minutes to reach us.
>>>> 
>>>> Hope they find the culprit and get it fixed quickly!
>>>> 
>>>> As of 3:27:45pm EDT their status page shows zero incidents
>>>> 
>>>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fstatus.bandwidt
>>>> h.com%2f&c=E,1,0374XKPpekgcDcgZahsM1KMmqs8G8yvMF9tnfZpRJpXXTxrgL2-V
>>>> igndLyRQtrrMJGWJ6i1KaXs_XiCvMeXdhv1lhddgpQqt4YCMx53FElYAtdl7Uq2rzdD
>>>> ogA,,&typo=1
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%
>>> 2fmailman%2flistinfo%2fvoiceops&c=E,1,d6hh1sPNldOI_pqt_AGZFGNoKoWaZU
>>> bc84nufxrulb109riOKTWRby6yDeUYeLJQ0GbVnCNv0oy1_Tzugdlt5gWrQyeF7siUkS
>>> yZdGoiE5g,&typo=1
>>> 
>> 
>> ---------------------------------------------------------------------------
>> Peter Beckman                                                  Internet Guy
>> beckman at angryox.com                                https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.angryox.com%2f&c=E,1,rbn4pLGIqtePoufgSG0gpI7tx3qNUxYkzxKybtyGo25HFwBHxTfqzDSJg1H_qmBIXOjOH_b4tu1C5cz96CBpsy7q6D1eiz4LWuisYlckAEDvry-sBDZubw,,&typo=1
>> ---------------------------------------------------------------------
>> ------
>> 
>
> ---------------------------------------------------------------------------
> Peter Beckman                                                  Internet Guy
> beckman at angryox.com                                https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.angryox.com%2f&c=E,1,h-PZ56YcwFzh2h6U3OBrBfL5rynQyYW5r0MlA8zMv4sNnAfqCU4pZ_9YgrauFWYKZLrjaKoPNUsWlQXj28km8xOd0dN7RvxhrdCBocSWFsdIUFHMOEjHWA,,&typo=1
> ----------------------------------------------------------------------
> -----
>

---------------------------------------------------------------------------
Peter Beckman                                                  Internet Guy
beckman at angryox.com                                https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.angryox.com%2f&c=E,1,lT8OseOtcOeZQiLP2hkAYErLoNRmyvW7M4KFJ56BvNCSlN7ikWFKZ4H7cNZ2c64nu08oBE-eFm4iaKsZItbUYlobkcJiVcEaTWCDGwUVNA,,&typo=1
---------------------------------------------------------------------------
_______________________________________________
VoiceOps mailing list
VoiceOps at voiceops.org
https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fpuck.nether.net%2fmailman%2flistinfo%2fvoiceops&c=E,1,8Cb0Xq6iM0kFLdX8GNjgCSSWWSI33AVhXAoOUeNqDwe6J7iEp6leFZ_rnR_bjvjme2lamRMwBbGXZUVmkvFI--HcGQNNDcKk_1ucUnaWm259fMr6OMkIil9K&typo=1