[outages] [VoiceOps] Fonality?

[Rich Walz]

I was sent this: I believe they are still working on a more in depth root cause

"
Hello everyone,

We had a service disruption today.  As of a few moments ago, our services are actively being restored and, with few exceptions, the outage we had today is over.  

First and foremost, this email is an apology for this widespread outage.  I know that as your business phone system provider, you depend upon the telephone for sales and service, and HUD to help you run your company, and that what happened today probably destroyed a lot of the trust that we built to date.  Despite our best efforts, we didn’t deliver the reliability we’ve been known for, and I’m sorry.

Now, on to the details, which I hope is the first step toward rebuilding that trust. 

What we know right now, which isn’t much but I expect more details later tonight, is that we had a failure of our DNS services.  DNS, or Domain Name System, is what translates names (like fonality.com) to the internet’s IP addresses, which are numbers that identify each computer, phone, and other device everywhere in the world.  Despite having multiple data centers, and multiple layers of redundancy within and between our hosting locations, something common to all of them - our DNS service - failed.  And as a result, just about everything else went down with it, because they all depend upon DNS so they can be “found” by the various components of our system.  This includes our voice carriers, our control panel, our HUD service, and even our own office communications (website, phones, HUD, and email) so it was difficult to even respond to you if you tried to reach us.    

Something so critical as DNS certainly has redundancies, but in this case and for a reason we’re still investigating, both the primary and redundant DNS systems failed.  You can be sure we’re going to do everything in our power to ensure that this type of global event can’t ever happen again - we will learn from this incident and will be better for it.

In the future, if you’d like to receive proactive updates, please visit our public system status site at http://trust.fonality.com or http://fonality.statuspage.io and sign up for alerts. That way you can be kept up to date proactively.

Please accept my most sincere apologies for the trouble today, and know that all of us here at Fonality are going to work hard to ensure this type of incident never happens again.

- David

""



Rich Walz
Sent from my iPhone

> On Sep 17, 2016, at 6:08 PM, chris via Outages <outages at outages.org> wrote:
> 
> Just curious, did anyone get an RFO or any explanation of what happened?
> 
> chris
> 
>> On Fri, Sep 16, 2016 at 1:30 PM, Aryn Nakaoka 808.356.2901 <anakaoka at trinet-hi.com> wrote:
>> http://www.isitdownrightnow.com/fonality.com.html
>> 
>> http://downdetector.com/status/fonality
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Aryn H. K. Nakaoka
>> anakaoka at trinet-hi.com
>> 
>> Direct: 808.356.2901
>> Fax : 808.356.2919
>> 
>> Tri-net Solutions
>> 518 Holokahana Lane Suite #200
>> Honolulu, HI 96817
>> http://www.trinet-hi.com
>> 
>> https://twitter.com/AlohaTone
>> 
>> Aloha Tone PBX  https://www.youtube.com/watch?v=96YWPY9wCeU
>> 
>> Aloha Tone (HA) High Availability http://youtu.be/rJsr4k0RBH8
>> 
>> CONFIDENTIALITY NOTICE:  The information contained in this email and any attachments may be privileged, confidential and protected from disclosure.  Any disclosure, distribution or copying of this email or any attachments by persons or entities other than the intended recipient is prohibited. If you have received this email in error, please notify the sender immediately by replying to the message and deleting this email and any attachments from your system. Thank you for your cooperation.
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Fri, Sep 16, 2016 at 7:08 AM, Adam Moss <amoss at flowroute.com> wrote:
>>> Getting tons of reports from customers that their Fonality systems have stopped working.
>>> 
>>> Anyone got any news on this?  Looks like their twitter is blowing up too.
>>> 
>>> -- 
>>> 
>>> ADAM MOSS
>>> 
>>> SUPPORT ENGINEER
>>> 
>>> 
>>> 
>>> +1.206.641.8034      
>>> 
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
>> 
>> 
>> _______________________________________________
>> VoiceOps mailing list
>> VoiceOps at voiceops.org
>> https://puck.nether.net/mailman/listinfo/voiceops
> 
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20160917/af3d0353/attachment.htm>