<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:564073923;
mso-list-type:hybrid;
mso-list-template-ids:1028541062 -1 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">So here is a <i>neat</i> little situation I ran into recently, and is worth sharing and reading; if this saves a life it was worth the crap I had to go through …..</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">== The Scenario ==</p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0in" type="disc">
<li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Expressway C/E 8.10.3 cluster over wan (2 Control Peers, 2 Edge Peers)</li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Customer deployed and managed SD-WAN solution in front of the Edge cluster to the Internet (with two separate transport carriers). I think it was Palos, but we’ll call it a whitebox’ed
solution for our purposes</li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">Using MRA and B2B Expressway configs</li><li class="MsoListParagraph" style="margin-left:0in;mso-list:l0 level1 lfo1">UAT for MRA and B2B is accepted and works great</li></ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">== The Problem ==</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The customer applies the zone/search rule config in Expressway for CMR and notices that randomly, during a presentation session in the CMR, the BFCP server (AKA, the WebEx meeting) will close the BFCP presentation to the endpoint coming
from the customer’s Expressway; all other BFCP clients are still receiving the BFCP presentation. That’s right, it
<i>appears</i> that WebEx <i>kicked</i> the BFCP participant coming from the customer’s Edge, but not because the BFCP server closed the session (all other participants remain)! Although it was happening randomly’ish in length of time into the presentation,
it would always happen at some point to the endpoint, generally around the 2 minute’ish mark.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">== The diagnosis ==</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Although random, a consistent’ish length would seem to suggest a timer / re-invite of some flavor, and that would be wrong, as ultimately uncovered. Sparing you all the gory tales of escalation and vendor bus underskirt sliding; the issue
was in fact, the SD-WAN solution itself. </p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">== The Explanation & The Fix ==</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What was happening is that every 120 seconds or so, the BFCP server (WebEx meeting) would send a UDP BFCP packet to all the BFCP presentation subscribers. The customer’s SD-WAN solution was
<i>identifying</i> these packets according to the customer (gotta love layer 7 capable firewalls
<span style="font-family:"Segoe UI Emoji",sans-serif">😊</span>) and queueing them onto a physically different link than which the stream was on, thus creating
<b>physical asymmetry, delay and latency</b>. I specifically requested that all inspection capabilities be turned off for the traffic but I guess that isn’t the same as “identifying the traffic” …. Lol. In a TCP stream, this would likely be tolerated to a degree
as packet loss or delay and/or jitter and would simply re transmit ….. but we are dealing with
<b>UDP</b> here, no bueno.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">To resolve, the customer had to identify and classify the traffic and force a active/failover transmission through the SD-WAN solution for that traffic, rather than a “load balance” transmission behavior.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">== Sleuthing & The Closing ==</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In hind sight, seems simple and makes perfect sense right? However, when your only visibility into the network is the Expressway servers themselves, it can be
<b>very</b> challenging to discover because at that point in the topology, everything looks like it is coming from and going to the VIP on the firewall pair. So how do you catch something like this when you can’t see everything?
<b>PCAPs</b>. <b>Literally counting f**king packet sequence numbers for 6 hours and identifying a consistent pattern of packets coming out of order and being “lost”.<o:p></o:p></b></p>
<p class="MsoNormal"><b><o:p> </o:p></b></p>
<p class="MsoNormal">-Ryan-</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>