[outages] VPN issues over Spectrum L3 boundaries

[Corey Davelaar]

I had a recent issue that was similar to this.  In that case it was a DDOS signature update on a specific vendors' DDOS scrubber at the host site that was the problem.  The specific tunnel src/dst flow would be dropped, however all other traffic between endpoints was allowed.  Because all the traffic for that specific flow hit a threshold above the rule and the IP's in question were flagged low enough in reputation to be fully inspected, it was blocked.  After whitelisting the IP's in the DDOS solution, all VPN traffic worked fine.

Just a thought.
________________________________
From: Outages <outages-bounces at outages.org> on behalf of Biddle, Josh via Outages <outages at outages.org>
Sent: Monday, October 21, 2019 12:39 PM
To: Marty Adkins <marty at martyadkins.com>; outages at outages.org <outages at outages.org>
Cc: D L <route2null0 at yahoo.com>; Cullis, Ben <BCullis at ntst.com>; Cochran, Brian <BCochran at ntst.com>
Subject: Re: [outages] VPN issues over Spectrum L3 boundaries


Our issues have recently magically resolved (last Thursday 10/17) due to L3 vanishing from the hop list. Our traffic now disappears into ntt.net and we are seeing two way IPsec traffic without any issues.



Marty, did your issues resolve?

[cid:image001.png at 01D58815.06155610]



From: Marty Adkins <marty at martyadkins.com>
Sent: Friday, October 18, 2019 4:01 PM
To: Biddle, Josh <JBiddle at ntst.com>; outages at outages.org
Subject: Re: [outages] VPN issues over Spectrum L3 boundaries



A customer of mine has had the same issue with a TW-connected site in Ohio and another in PA.  One VPN tunnel works fine, the other has one-way ISAKMP traffic to the other head-end, which connects to Level3.  A traceroute shows the failing path includes 66.109.7.162.  The failing direction is from the PA/OH sites toward the L3 head end.  Full-size pings work fine.  It's the UDP/500 that vanishes.  I.E., it has nothing to do with MTU.

Both started having the issue around Aug 26.  Mysteriously, every week to 10 days, the broken path will start working for a while.  This is usually shortly after midnight EDT; they go back down 1-3 hours later and stay down.  The log entries for the two sites match within seconds.

The customer's contract is with Comcast Business so it's been difficult to get to someone clueful about this symptom in TW.

-Marty

On 10/15/2019 8:04 AM, Biddle, Josh via Outages wrote:

Found a thread in the Spectrum forums talking about the issue finally – it was marked as resolved so I started a new one.



https://forums.timewarnercable.com/t5/Connectivity/Traffic-issues-at-66-109-7-162/m-p/164091#M53497<https://urldefense.proofpoint.com/v2/url?u=https-3A__forums.timewarnercable.com_t5_Connectivity_Traffic-2Dissues-2Dat-2D66-2D109-2D7-2D162_m-2Dp_164091-23M53497&d=DwMD-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=pXHvd5iI_J5DVYGMDKBSdUZl1iyilZUvR3oyvE3BJ0E&s=_90gCNY2Ln1XdqWtMNguWIRejhsdUxoLyvgTpo-R5jM&e=>



From: Outages <outages-bounces at outages.org><mailto:outages-bounces at outages.org> On Behalf Of Biddle, Josh via Outages
Sent: Sunday, October 13, 2019 12:00 PM
To: outages at outages.org<mailto:outages at outages.org>
Subject: [outages] VPN issues over Spectrum L3 boundaries



We have several offices over the Ohio and Pennsylvania area that are experiencing issues passing traffic over VPN tunnels (specifically, there is always a Spectrum >< Level 3 interconnect). It is a very strange issue. The VPN tunnel will actually establish, and if you source your ping from inside the internal network across the VPN tunnel to the destination, the traffic gets there and replies, but the replies never make it back to the original sending point.



Anyone else experiencing any similar issues like this?



Best Regards,

Josh





This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance at NTST.com immediately and permanently delete this email and any attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20191021/e244bbaf/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 169528 bytes
Desc: image001.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20191021/e244bbaf/attachment.png>