<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"helvetica neue";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Whaaaaaaaa...?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Do you mean that, in response to inbound UDP traffic, something was sending back, like, TCP RST? Or there was an actual payload inside with an English text string?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Was the source IP just the same as the IP you were targeting, or seemingly some carrier router in the middle?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Did you happen to save a pcap of these TCP packet(s) that were coming back to you? Because this I've gotta see...<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>If they're just sending back TCP RST, then I guess I could envision an *extremely* broken or misconfigured firewall doing something like that. Still...damn strange.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-- Nathan<o:p></o:p></span></p><p class=MsoNormal><a name="_MailEndCompose"><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></a></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> VoiceOps [mailto:voiceops-bounces@voiceops.org] <b>On Behalf Of </b>Kent A via VoiceOps<br><b>Sent:</b> Friday, March 8, 2024 8:51 AM<br><b>To:</b> voiceops@voiceops.org<br><b>Subject:</b> Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>ICMP I would have noticed right away. I thought we were screaming into the void until at last I found those packets. Something was clearly consuming my UDP, and replying with a rejection message in TCP, not ICMP. It made zero sense. Interestingly, the client just called and said all functionality has been restored.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>-Kent<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> VoiceOps <<a href="mailto:voiceops-bounces@voiceops.org">voiceops-bounces@voiceops.org</a>> <b>On Behalf Of </b>Nathan Anderson via VoiceOps<br><b>Sent:</b> Friday, March 8, 2024 11:47 AM<br><b>To:</b> <a href="mailto:voiceops@voiceops.org">voiceops@voiceops.org</a><br><b>Subject:</b> Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>"but I did get TCP rejections to my inbound UDP packets"<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Did you mean you got ICMP messages back from some router on their network in response to the UDP transmissions you were sending? Not TCP, surely...<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-- Nathan<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> VoiceOps [</span><a href="mailto:voiceops-bounces@voiceops.org"><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>mailto:voiceops-bounces@voiceops.org</span></a><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>] <b>On Behalf Of </b>Kent A via VoiceOps<br><b>Sent:</b> Friday, March 8, 2024 7:55 AM<br><b>To:</b> </span><a href="mailto:voiceops@voiceops.org"><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>voiceops@voiceops.org</span></a><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'><br><b>Subject:</b> Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>I too have a customer in the same area with the same exact problem. However they are doing SIP over UDP, and initially inbound SIP packets from my server headed to the client at frontier were getting rejections. I could see their SIP packets, but they never got mine, but I did get TCP rejections to my inbound UDP packets. Frontier rolled a truck, the tech said there was nothing he could do and he would escalate to Tier II. As soon as his truck left the parking lot, SIP was working in both directions, but this new RTP situation was now present. Never received a callback from Tier II, but it’s clear they were able to “unblock” SIP packets that they suddenly started blocking. Would love to know if a VPN gets around this whole mess. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>-Kent <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> VoiceOps <</span><a href="mailto:voiceops-bounces@voiceops.org"><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>voiceops-bounces@voiceops.org</span></a><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>> <b>On Behalf Of </b>Nathan Anderson via VoiceOps<br><b>Sent:</b> Friday, March 8, 2024 10:45 AM<br><b>To:</b> </span><a href="mailto:voiceops@voiceops.org"><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>voiceops@voiceops.org</span></a><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><br><b>Subject:</b> Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Assuming I understood the original problem description accurately, I don't see how a lossy trunk somewhere could explain 100% loss of *just* RTP and only in *one* specific direction with *zero* impact to any other internet traffic. You'd think that at least some RTP frames would manage to squeak by every once in a while, and that these users would have complaints about other non-voice-related things also not working as well.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Of course, the TLS would only make it impossible for the RXing carrier to peer into the SIP signalling to see details about the RTP session set-up. So if they're doing something more brain-dead to UDP in general, TLSing the SIP isn't going to work around that. Taking one of your affected customers and encapsulating the whole enchilada -- SIP, RTP, and all -- within a VPN would actually be a pretty interesting experiment...<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>-- Nathan<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> VoiceOps [</span><a href="mailto:voiceops-bounces@voiceops.org"><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>mailto:voiceops-bounces@voiceops.org</span></a><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>] <b>On Behalf Of </b>Jim Rodgers via VoiceOps<br><b>Sent:</b> Friday, March 8, 2024 7:09 AM<br><b>To:</b> Mike Hammett<br><b>Cc:</b> </span><a href="mailto:voiceops@voiceops.org"><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>voiceops@voiceops.org</span></a><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'><br><b>Subject:</b> Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>We use SIP TLS. I don't think they're intentionally blocking voice traffic, I think there's something broken inside their network that needs to be fixed (lossy link somewhere?).<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>The issue is getting anyone there to recognize the issue and want to fix it.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Jim<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>On Fri, Mar 8, 2024 at 5:32 AM Mike Hammett <<a href="mailto:voiceops@ics-il.net">voiceops@ics-il.net</a>> wrote:<o:p></o:p></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><div><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>I don't trust last mile networks to reliably deliver SIP calls. I usually end up putting them into VPNs, TLS, etc.<o:p></o:p></span></p><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><br><br>-----<br>Mike Hammett<br>Intelligent Computing Solutions<br></span><a href="http://www.ics-il.com" target="_blank"><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>http://www.ics-il.com</span></a><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><br><br><br><br>Midwest Internet Exchange<br></span><a href="http://www.midwest-ix.com" target="_blank"><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>http://www.midwest-ix.com</span></a><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><div class=MsoNormal align=center style='text-align:center'><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><hr size=4 width="100%" align=center></span></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-family:"Helvetica","sans-serif";color:black'>From: </span></b><span style='font-family:"Helvetica","sans-serif";color:black'>"Jim Rodgers via VoiceOps" <</span><a href="mailto:voiceops@voiceops.org" target="_blank"><span style='font-family:"Helvetica","sans-serif"'>voiceops@voiceops.org</span></a><span style='font-family:"Helvetica","sans-serif";color:black'>><br><b>To: </b></span><a href="mailto:voiceops@voiceops.org" target="_blank"><span style='font-family:"Helvetica","sans-serif"'>voiceops@voiceops.org</span></a><span style='font-family:"Helvetica","sans-serif";color:black'><br><b>Sent: </b>Thursday, March 7, 2024 11:16:23 AM<br><b>Subject: </b>[VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)<o:p></o:p></span></p><div><div><div><p class=MsoNormal><span style='font-family:"Helvetica","sans-serif";color:black'>Beginning early yesterday, we're seeing dropped voice rtp traffic to some of our business customers in the Los Angeles metro area on Frontier Comm broadband fiber. The voice udp stream is leaving our data center and never making it to the Frontier fiber customer. It's not all of our customers, only random ones. We've sniffed the traffic on our side and see the voice rtp stream leave our data center but then sniffing on our customer's side the traffic never arrives (multiple Frontier fiber customers with this issue, not just one).<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Helvetica","sans-serif";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Helvetica","sans-serif";color:black'>Switching the customer over to an alternate Internet connection resolves the issue.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Helvetica","sans-serif";color:black'><o:p> </o:p></span></p></div><div><div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'>Frontier frontline customer support doesn’t get it and they just want to roll a tech out for an issue that’s deeper inside their network.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'>We have packet captures of both sides (our DC and your customer) showing the voice rtp stream leaving our DC and never showing up at the fiber customer.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'>This doesn’t seem to be affecting every fiber customer in the Frontier footprint, it just seems to be random customers.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'>Anyone else experiencing this issue? Any thoughts on who to contact on the Frontier side to get it resolved and/or get some eyes on it?<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'>Thanks for the help.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"helvetica neue";color:black'>Jim<o:p></o:p></span></p></div></div></div></div></div><p class=MsoNormal><span style='font-family:"Helvetica","sans-serif";color:black'><br>_______________________________________________<br>VoiceOps mailing list<br></span><a href="mailto:VoiceOps@voiceops.org" target="_blank"><span style='font-family:"Helvetica","sans-serif"'>VoiceOps@voiceops.org</span></a><span style='font-family:"Helvetica","sans-serif";color:black'><br></span><a href="https://puck.nether.net/mailman/listinfo/voiceops" target="_blank"><span style='font-family:"Helvetica","sans-serif"'>https://puck.nether.net/mailman/listinfo/voiceops</span></a><span style='font-family:"Helvetica","sans-serif";color:black'><o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p></div></div></div></blockquote></div></div></body></html>