<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div><span></span></div><div><meta http-equiv="content-type" content="text/html; charset=utf-8"><div><div style="direction: inherit;">I was sent this: I believe they are still working on a more in depth root cause</div><div style="direction: inherit;"><br></div><div style="direction: inherit;">"</div><div style="direction: inherit;"><p style="margin-bottom: 1em;"><span style="background-color: rgba(255, 255, 255, 0);">Hello everyone,<br><span color="#5856d6"><br></span>We had a service disruption today. As of a few moments ago, our services are actively being restored and, with few exceptions, the outage we had today is over. <br><span color="#5856d6"><br></span>First and foremost, this email is an apology for this widespread outage. I know that as your business phone system provider, you depend upon the telephone for sales and service, and HUD to help you run your company, and that what happened today probably destroyed a lot of the trust that we built to date. Despite our best efforts, we didn’t deliver the reliability we’ve been known for, and I’m sorry.<br><span color="#5856d6"><br></span>Now, on to the details, which I hope is the first step toward rebuilding that trust. <br><span color="#5856d6"><br></span>What we know right now, which isn’t much but I expect more details later <span class="aBn" data-term="goog_663684285" tabindex="0">tonight</span>, is that we had a failure of our DNS services. DNS, or Domain Name System, is what translates names (like <a href="http://www.fonality.com/e1t/c/*W2t5sft6LxQ6LW9bd05t5-sMR40/*VvSNVC8mDQ7lW3hW6SB1VTsbb0/5/f18dQhb0S8388XJ92MW6_T8vK2YX2vxW7bd_Hx7N_xp5W74C3tk2zHMlpW5tsJhd4Dl6HzW3LRnwR49X0xTW4Bs6cn4yx_8pW2-J-SL2z_tPYN41Q2WgZTf7NW1YTF-H7TnHDPW5L1yJ85FxwrJVXjfSK1GgGq8W1YYV5x72nt_yW6sxy0G7JWL5mW7BWSYd1hMwMsW1Ggvq67lk60HW7scvx42bPk3YW1nwdD67vZk9pW7CSg4b5XX0NhW82mHhc1Yk8PmW1GczZL6PJjY8W795yf37qgqfZW6sQvTk22RSx_W5YV9WL6Bq1pyW6WqMZJ61fz3TW3G_1lT6l4V95N20SZDScZLWRW86l68q8fbMtzW5K6Yp66W4F7WW7bjbYM5gXyLYW61lPZv25qk-mW3srzzb6B5sgxVbtxKm6QSkZ4W4ZQ-sf1qyTQXW9dCq_x4Px9HJM5XRZt-f1W5102" target="_blank" data-saferedirecturl="https://www.google.com/url?hl=en&q=http://fonality.com/&source=gmail&ust=1474145827844000&usg=AFQjCNG6Nlil3D5uYouZhEZ8dconlqtmVQ" data-hs-link-id="0">fonality.com</a>) to the internet’s IP addresses, which are numbers that identify each computer, phone, and other device everywhere in the world. Despite having multiple data centers, and multiple layers of redundancy within and between our hosting locations, something common to all of them - our DNS service - failed. And as a result, just about everything else went down with it, because they all depend upon DNS so they can be “found” by the various components of our system. This includes our voice carriers, our control panel, our HUD service, and even our own office communications (website, phones, HUD, and email) so it was difficult to even respond to you if you tried to reach us. <br><span color="#5856d6"><br></span>Something so critical as DNS certainly has redundancies, but in this case and for a reason we’re still investigating, both the primary and redundant DNS systems failed. You can be sure we’re going to do everything in our power to ensure that this type of global event can’t ever happen again - we will learn from this incident and will be better for it.<br><span color="#5856d6"><br></span>In the future, if you’d like to receive proactive updates, please visit our public system status site at <a href="http://www.fonality.com/e1t/c/*W2t5sft6LxQ6LW9bd05t5-sMR40/*W4QT6pX4g_-lYW8fRLjk3pBGL-0/5/f18dQhb0S83c8XJ8g4W6_T8vK2YX2vxW7bd_Hx7N_xp5W74C5B52zHMlpW5tsJgX2-F7BmW3s91783B-9l9W62hsLx6nPlPpW2HVTVL4DqHqzW6dgmw34yv2r0W3MlV2W1Df0XLW6yBPHz1ddcKhW3bPyFL3BJtYDW5Wd0bF5-2sjBW7ZzDyK7GLjPjW2KrqQ_7Pw_HgW7H0KG61nCLt2W2g0jln7Ys37HW6XfHpJ76vjg_W24NJtk78_ldxW7PM2p220W74bW1HsRNk2b2_4BW1L2qYd6cQ74MW7k4lBK1lr5k2W82q_rv1rmt1JW22MgMR7WM3QpW7nTXXS22KRNkW42ktnR8ZsTwkVcWbVf4Wpz-0W3Xt-ws8V02zzW4ZY-ls79T9fbN4NcyPB84rVzW6LzXZn9jN4qkN46JVyLpW0pjW7zq_5w7LsgrtW8DYpNY4x463tW5t-_-b2sc1tgW1g7MQP1qZL5T0" target="_blank" data-saferedirecturl="https://www.google.com/url?hl=en&q=http://trust.fonality.com/&source=gmail&ust=1474145827845000&usg=AFQjCNFsQx1FubAzrvNgKpsOBZ2SXEOWmg" data-hs-link-id="0">http://trust.fonality.com</a> or <a href="http://www.fonality.com/e1t/c/*W2t5sft6LxQ6LW9bd05t5-sMR40/*W4PdstR3CcqsTW8v_mXl6jkyST0/5/f18dQhb0S83f8XJ9j-W6_T8vK2YX2vxW7bd_Hx7N_xp5W74C4JJ2zHMlpW5tsJhd4Dl6HzW3LRnwR47xh8SW3T2wdz4v1zp8W6-m4Sm2HVTVLW4DqHqz6dgmw3W4yv2r03MlV2WW1Df0XL6yBPHzW1ddcKh3bPyFLN3CbfH6ZgnVgW1GgGq81YYV5xW72nt_y6sxy0GW6wDfmh699J3pW7NWM6d2d1PChW7TQ1Nj6hYhK8W6MCX4x7MMvdsW1T9cLQ7BT_VQW717FvF7n0vT3W82qS-92dGGffW7QZCKQ6LwCzzW5XW9qw76vTl8W2d1wQ17dk5sMW85Mfkk7vX9lnW7v2Ftm3G_1lTW6m3-PQ41JXfHVrX3l16VzNX2W75k4zy256ykhW4r3pqp51vy4SN4vgKM142cYPW3nNLHX9kYtFTN24lqNgbZ08hW8sNr1R3TLlq9W8_wY7d6XCKKSW2M-NLJ5WGdW8W3-2pb81FQ06-0" target="_blank" data-saferedirecturl="https://www.google.com/url?hl=en&q=http://fonality.statuspage.io/&source=gmail&ust=1474145827845000&usg=AFQjCNFbYsZO2EWsctcWLm-Pb1PsRaA9HA" data-hs-link-id="0">http://fonality.statuspage.io</a> <wbr>and sign up for alerts. That way you can be kept up to date proactively.<br><span color="#5856d6"><br></span>Please accept my most sincere apologies for the trouble today, and know that all of us here at Fonality are going to work hard to ensure this type of incident never happens again.<br><span color="#5856d6"><br></span>- David</span></p><p style="direction: inherit; margin-bottom: 1em;"><span style="background-color: rgba(255, 255, 255, 0);">""</span></p></div><div style="direction: inherit;"><br></div><br>Rich Walz<div>Sent from my iPhone</div></div><div><br>On Sep 17, 2016, at 6:08 PM, chris via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Just curious, did anyone get an RFO or any explanation of what happened?<div><br></div><div>chris</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 16, 2016 at 1:30 PM, Aryn Nakaoka 808.356.2901 <span dir="ltr"><<a href="mailto:anakaoka@trinet-hi.com" target="_blank">anakaoka@trinet-hi.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><a href="http://www.isitdownrightnow.com/fonality.com.html" target="_blank">http://www.isitdownrightnow.<wbr>com/fonality.com.html</a><br><div><br></div><div><a href="http://downdetector.com/status/fonality" target="_blank">http://downdetector.com/<wbr>status/fonality</a><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div data-smartmail="gmail_signature"><br><br><br><br>Aryn H. K. Nakaoka<br><a href="mailto:anakaoka@trinet-hi.com" target="_blank">anakaoka@trinet-hi.com</a><br><br>Direct: <a href="tel:808.356.2901" value="+18083562901" target="_blank">808.356.2901</a><br>Fax : <a href="tel:808.356.2919" value="+18083562919" target="_blank">808.356.2919</a><br><br>Tri-net Solutions<br>518 Holokahana Lane Suite #200<br>Honolulu, HI 96817<br><a href="http://www.trinet-hi.com" target="_blank">http://www.trinet-hi.com</a><br><br><a href="https://twitter.com/AlohaTone" target="_blank">https://twitter.com/AlohaTone</a><br><br>Aloha Tone PBX <a href="https://www.youtube.com/watch?v=96YWPY9wCeU" target="_blank">https://www.youtube.com/<wbr>watch?v=96YWPY9wCeU</a><br><br>Aloha Tone (HA) High Availability <a href="http://youtu.be/rJsr4k0RBH8" target="_blank">http://youtu.be/rJsr4k0RBH8</a><br><br>CONFIDENTIALITY NOTICE: The information contained in this email and any attachments may be privileged, confidential and protected from disclosure. Any disclosure, distribution or copying of this email or any attachments by persons or entities other than the intended recipient is prohibited. If you have received this email in error, please notify the sender immediately by replying to the message and deleting this email and any attachments from your system. Thank you for your cooperation.<br><br><br><br><br><br></div></div>
<br><div class="gmail_quote"><div><div class="h5">On Fri, Sep 16, 2016 at 7:08 AM, Adam Moss <span dir="ltr"><<a href="mailto:amoss@flowroute.com" target="_blank">amoss@flowroute.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr">Getting tons of reports from customers that their Fonality systems have stopped working.<div><br></div><div>Anyone got any news on this? Looks like their twitter is blowing up too.</div><span><font color="#888888"><div><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><table width="480" bgcolor="#ffffff" cellspacing="0" cellpadding="0" style="font-family:Times;border-collapse:collapse"><tbody><tr><td valign="top" align="left" height="16"><div style="margin:0px;padding:0px;line-height:0"><img width="1" height="1" border="0" alt="" src="http://go.flowroute.com/l/48002/2015-05-06/qmxzf/48002/34618/201405_email_sign_empty.png" style="display:block"></div></td></tr><tr><td valign="top" align="left"><table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="0" style="border-collapse:collapse"><tbody><tr><td valign="top" align="left" width="336"><table width="100%" cellspacing="0" cellpadding="0" style="border-collapse:collapse"><tbody><tr><td valign="top" align="left" style="font-family:'Century Gothic',Arial,sans-serif;font-size:18px;color:rgb(55,55,55);font-weight:bold;line-height:14px;text-transform:uppercase">ADAM MOSS</td></tr><tr><td valign="top" align="left" height="8"><div style="margin:0px;padding:0px;line-height:0"><img width="1" height="1" border="0" alt="" src="http://go.flowroute.com/l/48002/2015-05-06/qmxzf/48002/34618/201405_email_sign_empty.png" style="display:block"></div></td></tr><tr><td valign="top" align="left" style="font-family:'Century Gothic',Arial,sans-serif;font-size:11px;color:rgb(55,55,55);line-height:11px;text-transform:uppercase">SUPPORT ENGINEER</td></tr></tbody></table></td><td width="144" valign="top" align="left"><div style="margin:0px;padding:0px"><a href="http://www.flowroute.com" target="_blank"><img width="144" height="30" border="0" alt="" src="http://go.flowroute.com/l/48002/2015-05-06/qmxzc/48002/34616/201405_email_sign_logo.png" style="display:block"></a></div></td></tr></tbody></table></td></tr><tr><td valign="top" align="left" height="12"><div style="margin:0px;padding:0px;line-height:0"><img width="1" height="1" border="0" alt="" src="http://go.flowroute.com/l/48002/2015-05-06/qmxzf/48002/34618/201405_email_sign_empty.png" style="display:block"></div></td></tr><tr><td valign="top" align="left"><table width="100%" cellspacing="0" cellpadding="0" style="border-collapse:collapse;border-top-width:5px;border-top-style:solid;border-top-color:rgb(240,245,245)"><tbody><tr><td valign="top" align="left" height="10"><div style="margin:0px;padding:0px;line-height:0"><img width="1" height="1" border="0" alt="" src="http://go.flowroute.com/l/48002/2015-05-06/qmxzf/48002/34618/201405_email_sign_empty.png" style="display:block"></div></td></tr><tr><td valign="top" align="left" style="font-family:'Century Gothic',Arial,sans-serif;font-size:12px;color:rgb(55,55,55);line-height:14px"><a href="tel:%2B1.206.641.8034" value="+12066418034" target="_blank">+1.206.641.8034</a> </td></tr></tbody></table></td></tr></tbody></table></div></div></div></div>
</div></font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
VoiceOps mailing list<br>
<a href="mailto:VoiceOps@voiceops.org" target="_blank">VoiceOps@voiceops.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/voiceops" rel="noreferrer" target="_blank">https://puck.nether.net/mailma<wbr>n/listinfo/voiceops</a><br>
<br></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
VoiceOps mailing list<br>
<a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/voiceops" rel="noreferrer" target="_blank">https://puck.nether.net/<wbr>mailman/listinfo/voiceops</a><br>
<br></blockquote></div><br></div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Outages mailing list</span><br><span><a href="mailto:Outages@outages.org">Outages@outages.org</a></span><br><span><a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a></span><br></div></blockquote></div></body></html>