<div dir="ltr"><div dir="ltr"><div dir="ltr">The issue seems to be with traffic going to Verizon. From all the traces we have seen (we looked from Level3, Cogent and others) the common denominator is br[2]3.nyc4. The traffic does not go through that device(s) then there isn't an issue. Also the issue seems to be with certain ports. So if you have one phone using say port 5062 and another using 5063 the one using 5063 may have the issue and the one using 5062 may not. You will most likely see the same devices having the issue over and over again, pulling it off the network for the NAT hole to close up may help since the next time it goes out it may use a different port. Over all it seems like they are doing some sort of hashing which is causing the traffic to go over a bad/overloaded card that is blackholing/dropping the packets.</div><div dir="ltr"><br></div><div>I spoke with a few people today from different organizations and they all said that they would be reaching out to Verizon. If anyone has the ear of an engineer at VZ please let me know. I have a few PCAP's clearly showing the issue along with many traces.</div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 13, 2019 at 6:21 PM Craig Haddow <<a href="mailto:craig@corevox.com" target="_blank">craig@corevox.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Is there any followup to this issue? We have a couple of fios customers that were experiencing one way audio where they could not hear the caller. We have the recordings that indicate the caller audio is reaching our hosted platform and the audio is good in both directions.<div><br><div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">—</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">Craig Haddow | CoreVox <br>Business Development Mgr</div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none">+1 215-390-1100 ext 102<br></div><br class="gmail-m_-5930661321359025184gmail-m_4856730503093873548Apple-interchange-newline" style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><span><span><img id="gmail-m_-5930661321359025184gmail-m_485673050309387354815BF9242-46D1-4C8E-9C0E-6C7E6D3D1F3C" src="cid:16ab34a45c4c58e0061"></span><br class="gmail-m_-5930661321359025184gmail-m_4856730503093873548Apple-interchange-newline"><span style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline"> <a href="https://www.corevox.com" target="_blank">https://www.corevox.com</a></span>
</span></span></div>
</div></div></div></div></div><div><br><blockquote type="cite"><div>On May 13, 2019, at 2:21 PM, Dovid Bender via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:</div><br class="gmail-m_-5930661321359025184gmail-m_4856730503093873548Apple-interchange-newline"><div><div dir="ltr">Izzy,<div><br></div><div>Please see my email on the nanog list. A few others reached out to me directly. I will call you in 5.</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 13, 2019 at 2:19 PM Izzy Goldstein - TeleGo via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Many of our clients who are using fios are reporting issues with their voip phones</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">when i switch them to their backup ISP (Optimum etc) it works fine</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default" style="font-family:monospace,monospace">anyone else experiencing any fios issues ?</div><div><br></div>-- <br><div dir="ltr" class="gmail-m_-5930661321359025184gmail-m_4856730503093873548gmail-m_8740036161820640416gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="font-size:12.8px"><div style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.3333px;font-family:arial;color:rgb(67,67,67);background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Izzy Goldstein </span></div><div style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.3333px;font-family:arial;color:rgb(67,67,67);background-color:transparent;font-style:italic;vertical-align:baseline;white-space:pre-wrap">Chief Technology Officer </span></div><div style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.3333px;font-family:arial;color:rgb(67,67,67);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Main: </span><span style="font-size:13.3333px;font-family:arial;color:rgb(17,85,204);background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><a href="tel:(212)%20477-1000" value="+12124771000" style="color:rgb(17,85,204)" target="_blank">(212) 477-1000 x 2085</a></span><span style="font-size:13.3333px;font-family:arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span></div><div style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.3333px;font-family:arial;color:rgb(67,67,67);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Direct: </span><span style="font-size:13.3333px;font-family:arial;color:rgb(17,85,204);background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><a href="tel:(929)%20477-2085" value="+19294772085" style="color:rgb(17,85,204)" target="_blank">(929) 477-2085</a></span></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:12.8px;font-style:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);line-height:1.2;margin-top:0pt;margin-bottom:0pt"><span style="font-size:13.3333px;font-family:arial;color:rgb(67,67,67);background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Website:</span><span style="font-size:13.3333px;font-family:arial;color:rgb(17,85,204);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span><a href="http://www.telego.net/" style="color:rgb(17,85,204);text-decoration:none" target="_blank"><span style="font-size:13.3333px;font-family:arial;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">www.telego.com</span></a><span style="font-size:13.3333px;font-family:arial;color:rgb(67,67,67);background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> </span></div><div style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12.8px"></span><span style="font-size:12.8px"><br></span></div><div style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><a href="http://www.telego.com/" style="color:rgb(17,85,204);text-decoration:none" target="_blank"><span style="font-size:13.3333px;font-family:arial;background-color:transparent;font-weight:700;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap"><img src="https://lh4.googleusercontent.com/Ze7NkT2QtotXfVWbpWXrpSMRIcAdqb0Ci0_RZsAjQzmRrd-_v02-O86QlqwZnGUE8SlBECXBvJUPXSH7ZRyDvDVeaXQFSS5SiA4gan112DN-0s2D3prRdt2UeUprZQEo1YmUFfZ7" width="182" height="54" style="border: none;"></span></a></div><div style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12.8px"><br></span></div><div style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:10.6667px;font-family:arial;color:rgb(153,153,153);font-style:italic;vertical-align:baseline;white-space:pre-wrap">Confidentiality Notice: This e-mail may contain confidential and/or privileged information. If you are not the intended recipient or have received this e-mail in error please notify us immediately by email reply and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of TeleGo (T). Employees of TeleGo are expressly required not to make defamatory statements and not to infringe or authorize any infringement of copyright or any other legal right by email communications. Any such communication is contrary to TeleGo policy and outside the scope of the employment of the individual concerned. TeleGo will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.</span></div><div style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:12.8px"><br></span><a href="https://youtu.be/DaT8YAZ4V0w" style="color:rgb(17,85,204);font-size:12.8px;text-decoration:none" target="_blank"><span style="font-size:14.6667px;font-family:arial;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">TeleGo Hosted PBX</span></a><br></div></span></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div>
_______________________________________________<br>Outages mailing list<br><a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br></div></blockquote></div><br></div></div></blockquote></div>