<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body bgcolor=white lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I confirmed with one customer that at least some of the calls they received during the second outage completed successfully. I would surmise that those calls would be the ones that I did see come in from the DR. But definitely nowhere near 100% of the call. Probably closer to 50% or less. (Though in fairness I should probably go through and check the ratio of calls received from both during that window; the customer's perception of percentages of failed calls could be skewed by the fact that they were surely getting a bunch of duplicate INVITEs as well.)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-- Nathan<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> VoiceOps [mailto:voiceops-bounces@voiceops.org] <b>On Behalf Of </b>Nate Burke<br><b>Sent:</b> Thursday, March 17, 2016 3:45 PM<br><b>To:</b> voiceops@voiceops.org<br><b>Subject:</b> Re: [VoiceOps] VoIP Innovations reliability<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'>I was seeing similar.<o:p></o:p></p><div><p class=MsoNormal>On 3/17/2016 5:43 PM, Nathan Anderson wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I did see some traffic come in from DR during that last outage, but when I placed test calls in from my cell, one call never completed ("the number you are dialing is not available") and the INVITE for that call definitely did not come in from the DR.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-- Nathan</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> VoiceOps [<a href="mailto:voiceops-bounces@voiceops.org">mailto:voiceops-bounces@voiceops.org</a>] <b>On Behalf Of </b>Nate Burke<br><b>Sent:</b> Thursday, March 17, 2016 3:27 PM<br><b>To:</b> <a href="mailto:voiceops@voiceops.org">voiceops@voiceops.org</a><br><b>Subject:</b> Re: [VoiceOps] VoIP Innovations reliability</span><o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'>Annnd they're down again.<o:p></o:p></p><div><p class=MsoNormal>On 3/17/2016 5:14 PM, Nate Burke wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal style='margin-bottom:12.0pt'>6 calls from 4 different CID numbers. All within 3 minutes. <o:p></o:p></p><div><p class=MsoNormal>On 3/17/2016 4:58 PM, Nathan Anderson wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Yesterday I definitely saw calls coming from the DR IP in my logs, but I had not yet added that IP as a peerin our SBC. I'll have to comb through logs today to see if we got any.</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Are you saying that the multiple calls you saw coming to your desk were all from the same number? If I had to guess, their side probably sent continuous INVITEs to you when it failed to get back an OK for any of them (not that you weren't sending back OK, but that their either didn't reach their SBC or did not reach it in a timely manner).</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-- Nathan</span><o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><o:p></o:p></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> VoiceOps [<a href="mailto:voiceops-bounces@voiceops.org">mailto:voiceops-bounces@voiceops.org</a>] <b>On Behalf Of </b>Jeff Waddell<br><b>Sent:</b> Thursday, March 17, 2016 2:53 PM<br><b>To:</b> Nate Burke<br><b>Cc:</b> <a href="mailto:voiceops@voiceops.org">voiceops@voiceops.org</a><br><b>Subject:</b> Re: [VoiceOps] VoIP Innovations reliability</span><o:p></o:p></p></div><p class=MsoNormal> <o:p></o:p></p><div><p class=MsoNormal>That is the issue a lot of our customers are reporting - where multiple calls are sent.... <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p><div><p class=MsoNormal>On Thu, Mar 17, 2016 at 5:49 PM, Nate Burke <<a href="mailto:nate@blastcomm.com" target="_blank">nate@blastcomm.com</a>> wrote:<o:p></o:p></p><div><p class=MsoNormal>I didn't see any traffic increment on the DR IP Address in my firewall rules, but this was odd. During the 15 minute period, I had probably 5 or 6 simultaneous calls ring into my desk. I normally only take a handfull of calls a day. <o:p></o:p></p><div><div><p class=MsoNormal> <o:p></o:p></p><div><p class=MsoNormal>On 3/17/2016 4:39 PM, Jeff Waddell wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>We implemented it too - I haven't checked to see if any traffic was sent across it <o:p></o:p></p><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div></div><div><p class=MsoNormal> <o:p></o:p></p><div><p class=MsoNormal>On Thu, Mar 17, 2016 at 5:20 PM, Nate Burke <<a href="mailto:nate@blastcomm.com">nate@blastcomm.com</a>> wrote:<o:p></o:p></p><div><p class=MsoNormal>Only 15 Minutes this time though. I had implemented the Disaster Recover Trunk as mentioned previously, but I didn't seem to be getting any calls completed through it. <o:p></o:p></p><div><div><p class=MsoNormal style='margin-bottom:12.0pt'> <o:p></o:p></p><div><p class=MsoNormal>On 3/17/2016 4:16 PM, Shripal Daphtary wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-bottom:12.0pt'>Down again! <o:p></o:p></p><div><p class=MsoNormal>Thanks, <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><p class=MsoNormal>Shripal<o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><br>On Mar 16, 2016, at 9:50 AM, Nate Burke <<a href="mailto:nate@blastcomm.com">nate@blastcomm.com</a>> wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-bottom:12.0pt'>Looks like it just came back up for me. Just over 30 min.<br><br>Nate<o:p></o:p></p><div><p class=MsoNormal>On 3/16/2016 8:45 AM, Shripal Daphtary wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-bottom:12.0pt'>We are experiencing an outage as well. <o:p></o:p></p><div><p class=MsoNormal>Thanks, <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><p class=MsoNormal>Shripal<o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><br>On Mar 16, 2016, at 9:36 AM, Nate Burke <<a href="mailto:nate@blastcomm.com">nate@blastcomm.com</a>> wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal style='margin-bottom:12.0pt'>Problems again this morning? Looks to be acting the same as it has been. <o:p></o:p></p><div><p class=MsoNormal>On 3/11/2016 6:00 PM, Alexander Lopez wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>I added them to our monitoring platform, stated getting alarms this past hour or so. <o:p></o:p></p><div><p class=MsoNormal>Up and down.<o:p></o:p></p></div><p class=MsoNormal style='margin-bottom:12.0pt'><br><br>-------- Original message --------<br>From: Nathan Anderson <a href="mailto:nathana@fsr.com"><nathana@fsr.com></a> <br>Date: 3/11/2016 6:31 PM (GMT-05:00) <br>To: 'Nate Burke' <a href="mailto:nate@blastcomm.com"><nate@blastcomm.com></a>, <a href="mailto:voiceops@voiceops.org">voiceops@voiceops.org</a> <br>Subject: Re: [VoiceOps] VoIP Innovations reliability <o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt'>...aaaaaand we're back.<br><br>-- Nathan<br><br>-----Original Message-----<br>From: VoiceOps [<a href="mailto:voiceops-bounces@voiceops.org">mailto:voiceops-bounces@voiceops.org</a>] On Behalf Of Nathan Anderson<br>Sent: Friday, March 11, 2016 3:30 PM<br>To: 'Nate Burke'; <a href="mailto:voiceops@voiceops.org">voiceops@voiceops.org</a><br>Subject: Re: [VoiceOps] VoIP Innovations reliability<br><br>It *feels* like they are under attack again, since I get a response to a ping once every 20 or so.<br><br>-- Nathan<br><br>-----Original Message-----<br>From: VoiceOps [<a href="mailto:voiceops-bounces@voiceops.org">mailto:voiceops-bounces@voiceops.org</a>] On Behalf Of Nathan Anderson<br>Sent: Friday, March 11, 2016 3:28 PM<br>To: 'Nate Burke'; <a href="mailto:voiceops@voiceops.org">voiceops@voiceops.org</a><br>Subject: Re: [VoiceOps] VoIP Innovations reliability<br><br>Confirmed.<br><br>-- Nathan<br><br>-----Original Message-----<br>From: VoiceOps [<a href="mailto:voiceops-bounces@voiceops.org">mailto:voiceops-bounces@voiceops.org</a>] On Behalf Of Nate Burke<br>Sent: Friday, March 11, 2016 3:26 PM<br>To: <a href="mailto:voiceops@voiceops.org">voiceops@voiceops.org</a><br>Subject: Re: [VoiceOps] VoIP Innovations reliability<br><br>Anyone else show them down again right now? My traceroutes aren't even <br>leaving Chicago. Dying at a Chicago hop on Level3.<br><br>Nate<br><br>On 3/6/2016 6:50 PM, Nathan Anderson wrote:<br>> Did anybody else just suffer another 45-minute-ish long outage from about 4:00p PST to 4:45p PST (ending about 5 minutes ago)?<br>><br>> -- Nathan<br>><br>> -----Original Message-----<br>> From: <a href="mailto:frnkblk@iname.com">frnkblk@iname.com</a> [<a href="mailto:frnkblk@iname.com">mailto:frnkblk@iname.com</a>]<br>> Sent: Sunday, March 06, 2016 4:43 PM<br>> To: Nathan Anderson; <a href="mailto:voiceops@voiceops.org">voiceops@voiceops.org</a><br>> Subject: RE: VoIP Innovations reliability<br>><br>> More here: <a href="http://blog.voipinnovations.com/blog">http://blog.voipinnovations.com/blog</a><br>><br>> Frank<br>><br>> -----Original Message-----<br>> From: VoiceOps [<a href="mailto:voiceops-bounces@voiceops.org">mailto:voiceops-bounces@voiceops.org</a>] On Behalf Of Nathan<br>> Anderson<br>> Sent: Tuesday, March 01, 2016 8:27 PM<br>> To: <a href="mailto:voiceops@voiceops.org">voiceops@voiceops.org</a><br>> Subject: [VoiceOps] VoIP Innovations reliability<br>><br>> Holy schlamoly. Anybody else use them here and being handed outage after<br>> outage over the last 2 days? Seriously thinking at this point about doing<br>> something else. This is ridiculous.<br>><br>> I desperately need sleep and if my cell goes off one more time...<br>><br>> -- Nathan<br>> _______________________________________________<br>> VoiceOps mailing list<br>> <a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><br>> <a href="https://puck.nether.net/mailman/listinfo/voiceops">https://puck.nether.net/mailman/listinfo/voiceops</a><br>><br>><br>><br>> _______________________________________________<br>> VoiceOps mailing list<br>> <a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><br>> <a href="https://puck.nether.net/mailman/listinfo/voiceops">https://puck.nether.net/mailman/listinfo/voiceops</a><br><br>_______________________________________________<br>VoiceOps mailing list<br><a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><br><a href="https://puck.nether.net/mailman/listinfo/voiceops">https://puck.nether.net/mailman/listinfo/voiceops</a><br><br>_______________________________________________<br>VoiceOps mailing list<br><a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><br><a href="https://puck.nether.net/mailman/listinfo/voiceops">https://puck.nether.net/mailman/listinfo/voiceops</a><br><br>_______________________________________________<br>VoiceOps mailing list<br><a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><br><a href="https://puck.nether.net/mailman/listinfo/voiceops">https://puck.nether.net/mailman/listinfo/voiceops</a><br><br>_______________________________________________<br>VoiceOps mailing list<br><a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><br><a href="https://puck.nether.net/mailman/listinfo/voiceops">https://puck.nether.net/mailman/listinfo/voiceops</a></span><o:p></o:p></p></div></blockquote><p class=MsoNormal> <o:p></o:p></p></div></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>_______________________________________________<br>VoiceOps mailing list<br><a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><br><a href="https://puck.nether.net/mailman/listinfo/voiceops">https://puck.nether.net/mailman/listinfo/voiceops</a><o:p></o:p></p></div></blockquote></blockquote><p class=MsoNormal> <o:p></o:p></p></div></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>_______________________________________________<br>VoiceOps mailing list<br><a href="mailto:VoiceOps@voiceops.org" target="_blank">VoiceOps@voiceops.org</a><br><a href="https://puck.nether.net/mailman/listinfo/voiceops" target="_blank">https://puck.nether.net/mailman/listinfo/voiceops</a><o:p></o:p></p></div></blockquote></blockquote><p class=MsoNormal> <o:p></o:p></p></div></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><br>_______________________________________________<br>VoiceOps mailing list<br><a href="mailto:VoiceOps@voiceops.org" target="_blank">VoiceOps@voiceops.org</a><br><a href="https://puck.nether.net/mailman/listinfo/voiceops" target="_blank">https://puck.nether.net/mailman/listinfo/voiceops</a><o:p></o:p></p></div><p class=MsoNormal> <o:p></o:p></p></div></blockquote><p class=MsoNormal> <o:p></o:p></p></div></div></div><p class=MsoNormal style='margin-bottom:12.0pt'><br>_______________________________________________<br>VoiceOps mailing list<br><a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><br><a href="https://puck.nether.net/mailman/listinfo/voiceops" target="_blank">https://puck.nether.net/mailman/listinfo/voiceops</a><o:p></o:p></p></div><p class=MsoNormal> <o:p></o:p></p></div></blockquote><p class=MsoNormal><br><br><br><br><br><o:p></o:p></p><pre>_______________________________________________<o:p></o:p></pre><pre>VoiceOps mailing list<o:p></o:p></pre><pre><a href="mailto:VoiceOps@voiceops.org">VoiceOps@voiceops.org</a><o:p></o:p></pre><pre><a href="https://puck.nether.net/mailman/listinfo/voiceops">https://puck.nether.net/mailman/listinfo/voiceops</a><o:p></o:p></pre></blockquote><p class=MsoNormal> <o:p></o:p></p></blockquote><p class=MsoNormal><o:p> </o:p></p></div></body></html>