[Outages-discussion] [outages] Second Bandwidth Outage Sunday

[Charles Sprickman]

> On Sep 27, 2021, at 2:31 PM, Jared Geiger <jared at compuwizz.net> wrote:
> 
> It looks like Bandwidth dropped all their BGP peers except NTT, presumably to use NTT's DDoS services.

How many do they typically have?

> 
> As far as peering, Bandwidth offers connectivity via Megaport. Peerless (21808) is on a bunch of Equinix exchanges https://peeringdb.com/net/16256 <https://peeringdb.com/net/16256>. Inteliquent (40160) is on some exchanges as well https://peeringdb.com/net/18236 <https://peeringdb.com/net/18236>. 
> 
> I did try and get Peerless to join Coresite Any2East last year because they are in the same building but they said no.
> 
> On Mon, Sep 27, 2021 at 11:19 AM Douglas Vought <douglas.vought at themsgctr.com <mailto:douglas.vought at themsgctr.com>> wrote:
> They are. Several customers have contacted me demanding that I plug the phone line back in. 
> 
> On 9/27/2021 13:54, Drew Matthews via Outages wrote:
>> i believe they are having issues again, not sure if its ddos related. but they are experiencing some major issues. https://status.bandwidth.com/ <https://status.bandwidth.com/>
>> 
>> Update - Bandwidth teams are also investigating delays with Toll-Free Messaging at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions.
>> Sep 27, 13:01 EDT
>> Update - We continue to investigate this incident.
>> Sep 27, 11:57 EDT
>> Update - Bandwidth teams are also investigating delayed ports at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions.
>> Sep 27, 11:07 EDT
>> Update - We continue to investigate this incidnet.
>> Sep 27, 11:02 EDT
>> Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, 911 traffic, Portal & API with intermittent service disruption.
>> Sep 27, 10:00 EDT
>> Update - Bandwidth teams are investigating an incident impacting Voice, V2 Voice, and 911 traffic with intermittent service disruption.
>> Sep 27, 09:42 EDT
>> Investigating - Bandwidth teams are investigating an incident impacting our V2 Voice services. Customers may notice intermittent service disruptions.
>> Sep 27, 09:12 EDT
>> 
>> On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <outages at outages.org <mailto:outages at outages.org>> wrote:
>> Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at
>> 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at
>> Bandwidth. The impact to us was much reduced this time, as we saw some flapping
>> of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to
>> fall below our thresholds.
>> 
>> The API started acting up at 14:56:00 UTC, but recovered, and then has been
>> flapping between OK and degraded since then. As of 21:52 UTC, the API and
>> Portal/Dashboard are either non-responsive, slow, or degraded.
>> 
>> Inbound Voice Traffic has remaind at expected historical levels throughout.
>> I have not checked on Termination, but it was working earlier today without
>> issue.
>> 
>> Latest from BW https://status.bandwidth.com/ <https://status.bandwidth.com/>
>> 
>>      Update - Bandwidth teams have observed expected call behavior for 911
>>      calls. 911 calls are processing as designed and are fully restored. Teams
>>      continue to work on standard voice traffic.
>> 
>>      Sep 26, 17:20 EDT
>> 
>> I question the phrase "processing as designed" which implies it is not
>> working as the end user expects... odd.
>> 
>> Beckman
>> ---------------------------------------------------------------------------
>> Peter Beckman                                                  Internet Guy
>> beckman at angryox.com <mailto:beckman at angryox.com>                                https://www.angryox.com/ <https://www.angryox.com/>
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org <mailto:Outages at outages.org>
>> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
>> 
>> 
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org <mailto:Outages at outages.org>
>> https://puck.nether.net/mailman/listinfo/outages <https://puck.nether.net/mailman/listinfo/outages>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org <mailto:Outages-discussion at outages.org>
> https://puck.nether.net/mailman/listinfo/outages-discussion <https://puck.nether.net/mailman/listinfo/outages-discussion>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20210927/3a1b8976/attachment.htm>