[VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)

Fri Mar 8 12:01:19 EST 2024

Whaaaaaaaa...?

Do you mean that, in response to inbound UDP traffic, something was sending
back, like, TCP RST?  Or there was an actual payload inside with an English
text string?

Was the source IP just the same as the IP you were targeting, or seemingly some
carrier router in the middle?

Did you happen to save a pcap of these TCP packet(s) that were coming back to
you?  Because this I've gotta see...

If they're just sending back TCP RST, then I guess I could envision a *really*
broken or misconfigured firewall doing something like that.  Still...damn
strange.

-- Nathan

From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Kent A via
VoiceOps
Sent: Friday, March 8, 2024 8:51 AM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)

ICMP I would have noticed right away. I thought we were screaming into the void
until at last I found those packets. Something was clearly consuming my UDP,
and replying with a rejection message in TCP, not ICMP. It made zero sense.
Interestingly, the client just called and said all functionality has been
restored.

-Kent

From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Nathan Anderson via
VoiceOps
Sent: Friday, March 8, 2024 11:47 AM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)

"but I did get TCP rejections to my inbound UDP packets"

?

Did you mean you got ICMP messages back from some router on their network in
response to the UDP transmissions you were sending?  Not TCP, surely...

-- Nathan

From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Kent A via
VoiceOps
Sent: Friday, March 8, 2024 7:55 AM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)

I too have a customer in the same area with the same exact problem. However
they are doing SIP over UDP, and initially inbound SIP packets from my server
headed to  the client at frontier were getting rejections. I could see their
SIP packets, but they never got mine, but I did get TCP rejections to my
inbound UDP packets. Frontier rolled a truck, the tech said there was nothing
he could do and he would escalate to Tier II. As soon as his truck left the
parking lot, SIP was working in both directions, but this new RTP situation was
now present. Never received a callback from Tier II, but it’s clear they were
able to “unblock” SIP packets that they suddenly started blocking. Would love
to know if a VPN gets around this whole mess.

-Kent

From: VoiceOps <voiceops-bounces at voiceops.org> On Behalf Of Nathan Anderson via
VoiceOps
Sent: Friday, March 8, 2024 10:45 AM
To: voiceops at voiceops.org
Subject: Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)

Assuming I understood the original problem description accurately, I don't see
how a lossy trunk somewhere could explain 100% loss of *just* RTP and only in
*one* specific direction with *zero* impact to any other internet traffic. 
You'd think that at least some RTP frames would manage to squeak by every once
in a while, and that these users would have complaints about other
non-voice-related things also not working as well.

Of course, the TLS would only make it impossible for the RXing carrier to peer
into the SIP signalling to see details about the RTP session set-up.  So if
they're doing something more brain-dead to UDP in general, TLSing the SIP isn't
going to work around that.  Taking one of your affected customers and
encapsulating the whole enchilada -- SIP, RTP, and all -- within a VPN would
actually be a pretty interesting experiment...

-- Nathan

From: VoiceOps [mailto:voiceops-bounces at voiceops.org] On Behalf Of Jim Rodgers
via VoiceOps
Sent: Friday, March 8, 2024 7:09 AM
To: Mike Hammett
Cc: voiceops at voiceops.org
Subject: Re: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)

We use SIP TLS. I don't think they're intentionally blocking voice traffic, I
think there's something broken inside their network that needs to be fixed
(lossy link somewhere?).

The issue is getting anyone there to recognize the issue and want to fix it.

Jim

On Fri, Mar 8, 2024 at 5:32 AM Mike Hammett <voiceops at ics-il.net> wrote:

    I don't trust last mile networks to reliably deliver SIP calls. I usually
    end up putting them into VPNs, TLS, etc.

    -----
    Mike Hammett
    Intelligent Computing Solutions
    http://www.ics-il.com

    Midwest Internet Exchange
    http://www.midwest-ix.com

    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

    From: "Jim Rodgers via VoiceOps" <voiceops at voiceops.org>
    To: voiceops at voiceops.org
    Sent: Thursday, March 7, 2024 11:16:23 AM
    Subject: [VoiceOps] One Way Audio - Frontier Comm (Los Angeles area)

    Beginning early yesterday, we're seeing dropped voice rtp traffic to some
    of our business customers in the Los Angeles metro area on Frontier Comm
    broadband fiber. The voice udp stream is leaving our data center and never
    making it to the Frontier fiber customer. It's not all of our customers,
    only random ones. We've sniffed the traffic on our side and see the voice
    rtp stream leave our data center but then sniffing on our customer's side
    the traffic never arrives (multiple Frontier fiber customers with this
    issue, not just one).

    Switching the customer over to an alternate Internet connection resolves
    the issue.

    Frontier frontline customer support doesn’t get it and they just want to
    roll a tech out for an issue that’s deeper inside their network.

    We have packet captures of both sides (our DC and your customer) showing
    the voice rtp stream leaving our DC and never showing up at the fiber
    customer.

    This doesn’t seem to be affecting every fiber customer in the Frontier
    footprint, it just seems to be random customers.

    Anyone else experiencing this issue? Any thoughts on who to contact on the
    Frontier side to get it resolved and/or get some eyes on it?

    Thanks for the help.

    Jim

    _______________________________________________
    VoiceOps mailing list
    VoiceOps at voiceops.org
    https://puck.nether.net/mailman/listinfo/voiceops

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20240308/aecc88ae/attachment-0001.htm>