<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
It’s funny what we’ll do for “strategic customers”. Lol, ah well there was Jack and Coke at the end of that rainbow for sure :).<br>
<br>
<div id="AppleMailSignature">Sent from my iPhone</div>
<div><br>
On Apr 30, 2018, at 11:27, Anthony Holloway <<a href="mailto:avholloway+cisco-voip@gmail.com">avholloway+cisco-voip@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">Yes, what James said, thank you for sharing this info. I think I would have given up at "counting f**king packet sequence numbers."</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Mon, Apr 30, 2018 at 10:13 AM James Buchanan <<a href="mailto:james.buchanan2@gmail.com">james.buchanan2@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Painful as this was, hats off to you for writing this up and sharing. Much appreciated!</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Apr 30, 2018 at 3:36 PM, Ryan Huff <span dir="ltr">
<<a href="mailto:ryanhuff@outlook.com" target="_blank">ryanhuff@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="#954F72">
<div class="m_2136137545226500513m_-4420932210170331635WordSection1">
<p class="MsoNormal">So here is a <i>neat</i> little situation I ran into recently, and is worth sharing and reading; if this saves a life it was worth the crap I had to go through …..</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">== The Scenario ==</p>
<p class="MsoNormal"><u></u> <u></u></p>
<ul style="margin-top:0in" type="disc">
<li class="m_2136137545226500513m_-4420932210170331635MsoListParagraph" style="margin-left:0in">
Expressway C/E 8.10.3 cluster over wan (2 Control Peers, 2 Edge Peers)</li><li class="m_2136137545226500513m_-4420932210170331635MsoListParagraph" style="margin-left:0in">
Customer deployed and managed SD-WAN solution in front of the Edge cluster to the Internet (with two separate transport carriers). I think it was Palos, but we’ll call it a whitebox’ed solution for our purposes</li><li class="m_2136137545226500513m_-4420932210170331635MsoListParagraph" style="margin-left:0in">
Using MRA and B2B Expressway configs</li><li class="m_2136137545226500513m_-4420932210170331635MsoListParagraph" style="margin-left:0in">
UAT for MRA and B2B is accepted and works great</li></ul>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">== The Problem ==</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">The customer applies the zone/search rule config in Expressway for CMR and notices that randomly, during a presentation session in the CMR, the BFCP server (AKA, the WebEx meeting) will close the BFCP presentation to the endpoint coming
from the customer’s Expressway; all other BFCP clients are still receiving the BFCP presentation. That’s right, it
<i>appears</i> that WebEx <i>kicked</i> the BFCP participant coming from the customer’s Edge, but not because the BFCP server closed the session (all other participants remain)! Although it was happening randomly’ish in length of time into the presentation,
it would always happen at some point to the endpoint, generally around the 2 minute’ish mark.</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">== The diagnosis ==</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Although random, a consistent’ish length would seem to suggest a timer / re-invite of some flavor, and that would be wrong, as ultimately uncovered. Sparing you all the gory tales of escalation and vendor bus underskirt sliding; the issue
was in fact, the SD-WAN solution itself. </p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">== The Explanation & The Fix ==</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">What was happening is that every 120 seconds or so, the BFCP server (WebEx meeting) would send a UDP BFCP packet to all the BFCP presentation subscribers. The customer’s SD-WAN solution was
<i>identifying</i> these packets according to the customer (gotta love layer 7 capable firewalls
<span style="font-family:"Segoe UI Emoji",sans-serif">😊</span>) and queueing them onto a physically different link than which the stream was on, thus creating
<b>physical asymmetry, delay and latency</b>. I specifically requested that all inspection capabilities be turned off for the traffic but I guess that isn’t the same as “identifying the traffic” …. Lol. In a TCP stream, this would likely be tolerated to a degree
as packet loss or delay and/or jitter and would simply re transmit ….. but we are dealing with
<b>UDP</b> here, no bueno.</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">To resolve, the customer had to identify and classify the traffic and force a active/failover transmission through the SD-WAN solution for that traffic, rather than a “load balance” transmission behavior.</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">== Sleuthing & The Closing ==</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">In hind sight, seems simple and makes perfect sense right? However, when your only visibility into the network is the Expressway servers themselves, it can be
<b>very</b> challenging to discover because at that point in the topology, everything looks like it is coming from and going to the VIP on the firewall pair. So how do you catch something like this when you can’t see everything?
<b>PCAPs</b>. <b>Literally counting f**king packet sequence numbers for 6 hours and identifying a consistent pattern of packets coming out of order and being “lost”.<span class="m_2136137545226500513HOEnZb"><font color="#888888"><u></u><u></u></font></span></b></p>
<span class="m_2136137545226500513HOEnZb"><font color="#888888">
<p class="MsoNormal"><b><u></u> <u></u></b></p>
<p class="MsoNormal">-Ryan-</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</font></span></div>
</div>
<br>
_______________________________________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
<br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
</blockquote>
</div>
</div>
</blockquote>
</body>
</html>