<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Helvetica Neue";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">Most of our VOIP customers are complaining as well about the Bandwidth thing.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif">
</span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages-discussion <outages-discussion-bounces@outages.org>
<b>On Behalf Of </b>Eric Herron<br>
<b>Sent:</b> Monday, September 27, 2021 1:05 PM<br>
<b>To:</b> outages-discussion@outages.org<br>
<b>Subject:</b> Re: [Outages-discussion] [outages] Second Bandwidth Outage Sunday<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Multiple TACs have confirmed DDoS attacks on their network and they are continuing their mitigation efforts but it appears to be ongoing and intermittent.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My inbound/outbound has been working today but I have had DTMF not work at times throughout the day.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">It seems like you may get caught up in it or may not and a subset of services may be affected at any given time.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages <<a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a>>
<b>On Behalf Of </b>Drew Matthews via Outages<br>
<b>Sent:</b> Monday, September 27, 2021 1:54 PM<br>
<b>To:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Second Bandwidth Outage Sunday<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">i believe they are having issues again, not sure if its ddos related. but they are experiencing some major issues. <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__status.bandwidth.com_&d=DwMGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=BDTo3zD5mrHq76g_7qcoflUzLb6xwPZTKmhMmagZVNU&m=p_8LMWA3SliW7dnfE4HPOaFuo3Vag47Qwy6X4QPbBm4&s=Ue5ZN_IWcpkpEW0gFYkLcB90O1tEYf5NOydjybfz6h8&e=">https://status.bandwidth.com/</a><o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div style="margin-bottom:15.0pt;box-sizing:border-box;outline:0px">
<p class="MsoNormal" style="background:#FFFFF2;vertical-align:baseline"><b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Update</span></b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"> -
Bandwidth teams are also investigating delays with Toll-Free Messaging at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions.<br>
</span><span style="font-size:10.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Sep <var><span style="font-family:"Helvetica Neue";font-style:normal">27</span></var>, <var><span style="font-family:"Helvetica Neue";font-style:normal">13:01</span></var> EDT</span><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"><o:p></o:p></span></p>
</div>
<div style="margin-bottom:15.0pt;box-sizing:border-box;outline:0px">
<p class="MsoNormal" style="background:#FFFFF2;vertical-align:baseline"><b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Update</span></b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"> -
We continue to investigate this incident.<br>
</span><span style="font-size:10.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Sep <var><span style="font-family:"Helvetica Neue";font-style:normal">27</span></var>, <var><span style="font-family:"Helvetica Neue";font-style:normal">11:57</span></var> EDT</span><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"><o:p></o:p></span></p>
</div>
<div style="margin-bottom:15.0pt;box-sizing:border-box;outline:0px">
<p class="MsoNormal" style="background:#FFFFF2;vertical-align:baseline"><b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Update</span></b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"> -
Bandwidth teams are also investigating delayed ports at this time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with intermittent service disruptions.<br>
</span><span style="font-size:10.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Sep <var><span style="font-family:"Helvetica Neue";font-style:normal">27</span></var>, <var><span style="font-family:"Helvetica Neue";font-style:normal">11:07</span></var> EDT</span><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"><o:p></o:p></span></p>
</div>
<div style="margin-bottom:15.0pt;box-sizing:border-box;outline:0px">
<p class="MsoNormal" style="background:#FFFFF2;vertical-align:baseline"><b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Update</span></b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"> -
We continue to investigate this incidnet.<br>
</span><span style="font-size:10.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Sep <var><span style="font-family:"Helvetica Neue";font-style:normal">27</span></var>, <var><span style="font-family:"Helvetica Neue";font-style:normal">11:02</span></var> EDT</span><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"><o:p></o:p></span></p>
</div>
<div style="margin-bottom:15.0pt;box-sizing:border-box;outline:0px">
<p class="MsoNormal" style="background:#FFFFF2;vertical-align:baseline"><b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Update</span></b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"> -
Bandwidth teams are investigating an incident impacting Voice, V2 Voice, 911 traffic, Portal & API with intermittent service disruption.<br>
</span><span style="font-size:10.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Sep <var><span style="font-family:"Helvetica Neue";font-style:normal">27</span></var>, <var><span style="font-family:"Helvetica Neue";font-style:normal">10:00</span></var> EDT</span><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"><o:p></o:p></span></p>
</div>
<div style="margin-bottom:15.0pt;box-sizing:border-box;outline:0px">
<p class="MsoNormal" style="background:#FFFFF2;vertical-align:baseline"><b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Update</span></b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"> -
Bandwidth teams are investigating an incident impacting Voice, V2 Voice, and 911 traffic with intermittent service disruption.<br>
</span><span style="font-size:10.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Sep <var><span style="font-family:"Helvetica Neue";font-style:normal">27</span></var>, <var><span style="font-family:"Helvetica Neue";font-style:normal">09:42</span></var> EDT</span><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:#FFFFF2;vertical-align:baseline"><b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Investigating</span></b><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"> -
Bandwidth teams are investigating an incident impacting our V2 Voice services. Customers may notice intermittent service disruptions.<br>
</span><span style="font-size:10.0pt;font-family:"Helvetica Neue";color:#1C0A14;border:none windowtext 1.0pt;padding:0in">Sep <var><span style="font-family:"Helvetica Neue";font-style:normal">27</span></var>, <var><span style="font-family:"Helvetica Neue";font-style:normal">09:12</span></var> EDT</span><span style="font-size:12.0pt;font-family:"Helvetica Neue";color:#1C0A14"><o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">Again today, on the heels of the 2h 47m DDoS-related outage yesterday, right at<br>
11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another incident at<br>
Bandwidth. The impact to us was much reduced this time, as we saw some flapping<br>
of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive enough to<br>
fall below our thresholds.<br>
<br>
The API started acting up at 14:56:00 UTC, but recovered, and then has been<br>
flapping between OK and degraded since then. As of 21:52 UTC, the API and<br>
Portal/Dashboard are either non-responsive, slow, or degraded.<br>
<br>
Inbound Voice Traffic has remaind at expected historical levels throughout.<br>
I have not checked on Termination, but it was working earlier today without<br>
issue.<br>
<br>
Latest from BW <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__status.bandwidth.com_&d=DwMGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=BDTo3zD5mrHq76g_7qcoflUzLb6xwPZTKmhMmagZVNU&m=p_8LMWA3SliW7dnfE4HPOaFuo3Vag47Qwy6X4QPbBm4&s=Ue5ZN_IWcpkpEW0gFYkLcB90O1tEYf5NOydjybfz6h8&e=" target="_blank">
https://status.bandwidth.com/</a><br>
<br>
Update - Bandwidth teams have observed expected call behavior for 911<br>
calls. 911 calls are processing as designed and are fully restored. Teams<br>
continue to work on standard voice traffic.<br>
<br>
Sep 26, 17:20 EDT<br>
<br>
I question the phrase "processing as designed" which implies it is not<br>
working as the end user expects... odd.<br>
<br>
Beckman<br>
---------------------------------------------------------------------------<br>
Peter Beckman Internet Guy<br>
<a href="mailto:beckman@angryox.com" target="_blank">beckman@angryox.com</a>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__www.angryox.com_&d=DwMGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=BDTo3zD5mrHq76g_7qcoflUzLb6xwPZTKmhMmagZVNU&m=p_8LMWA3SliW7dnfE4HPOaFuo3Vag47Qwy6X4QPbBm4&s=LujDJI9ijucs-p7m3ncdeqRet5bz1KjUXVV6brFyllk&e=" target="_blank">
https://www.angryox.com/</a><br>
---------------------------------------------------------------------------<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages&d=DwMGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=BDTo3zD5mrHq76g_7qcoflUzLb6xwPZTKmhMmagZVNU&m=p_8LMWA3SliW7dnfE4HPOaFuo3Vag47Qwy6X4QPbBm4&s=Zs9LrL5PqbMvcLyMMgsLuHwYFI29rH4bVLGn7MMGano&e=" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>