[Outages-discussion] [outages] Second Bandwidth Outage Sunday

[Jared Geiger]

It looks like Bandwidth dropped all their BGP peers except NTT, presumably
to use NTT's DDoS services.

As far as peering, Bandwidth offers connectivity via Megaport. Peerless
(21808) is on a bunch of Equinix exchanges https://peeringdb.com/net/16256.
Inteliquent (40160) is on some exchanges as well
https://peeringdb.com/net/18236.

I did try and get Peerless to join Coresite Any2East last year because they
are in the same building but they said no.

On Mon, Sep 27, 2021 at 11:19 AM Douglas Vought <
douglas.vought at themsgctr.com> wrote:

> They are. Several customers have contacted me demanding that I plug the
> phone line back in.
> On 9/27/2021 13:54, Drew Matthews via Outages wrote:
>
> i believe they are having issues again, not sure if its ddos related. but
> they are experiencing some major issues. https://status.bandwidth.com/
>
> Update - Bandwidth teams are also investigating delays with Toll-Free
> Messaging at this time along with V2 Voice, Standard Voice, 911 traffic,
> Portal & API with intermittent service disruptions.
> Sep 27, 13:01 EDT
> Update - We continue to investigate this incident.
> Sep 27, 11:57 EDT
> Update - Bandwidth teams are also investigating delayed ports at this
> time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with
> intermittent service disruptions.
> Sep 27, 11:07 EDT
> Update - We continue to investigate this incidnet.
> Sep 27, 11:02 EDT
> Update - Bandwidth teams are investigating an incident impacting Voice,
> V2 Voice, 911 traffic, Portal & API with intermittent service disruption.
> Sep 27, 10:00 EDT
> Update - Bandwidth teams are investigating an incident impacting Voice,
> V2 Voice, and 911 traffic with intermittent service disruption.
> Sep 27, 09:42 EDT
> Investigating - Bandwidth teams are investigating an incident impacting
> our V2 Voice services. Customers may notice intermittent service
> disruptions.
> Sep 27, 09:12 EDT
>
> On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <
> outages at outages.org> wrote:
>
>> Again today, on the heels of the 2h 47m DDoS-related outage yesterday,
>> right at
>> 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another
>> incident at
>> Bandwidth. The impact to us was much reduced this time, as we saw some
>> flapping
>> of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive
>> enough to
>> fall below our thresholds.
>>
>> The API started acting up at 14:56:00 UTC, but recovered, and then has
>> been
>> flapping between OK and degraded since then. As of 21:52 UTC, the API and
>> Portal/Dashboard are either non-responsive, slow, or degraded.
>>
>> Inbound Voice Traffic has remaind at expected historical levels
>> throughout.
>> I have not checked on Termination, but it was working earlier today
>> without
>> issue.
>>
>> Latest from BW https://status.bandwidth.com/
>>
>>      Update - Bandwidth teams have observed expected call behavior for 911
>>      calls. 911 calls are processing as designed and are fully restored.
>> Teams
>>      continue to work on standard voice traffic.
>>
>>      Sep 26, 17:20 EDT
>>
>> I question the phrase "processing as designed" which implies it is not
>> working as the end user expects... odd.
>>
>> Beckman
>>
>> ---------------------------------------------------------------------------
>> Peter Beckman                                                  Internet
>> Guy
>> beckman at angryox.com
>> https://www.angryox.com/
>>
>> ---------------------------------------------------------------------------
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>
> _______________________________________________
> Outages mailing listOutages at outages.orghttps://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20210927/d3efce95/attachment.htm>