<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi, Ryan,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thank you very much for this detailed write up!<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Tom<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Ryan Huff [mailto:ryanhuff@outlook.com] <br><b>Sent:</b> Tuesday, October 27, 2015 3:32 PM<br><b>To:</b> Thomas LeMay; 'Jason Aarons (AM)'; 'Aaron Banks'; cisco-voip@puck.nether.net<br><b>Subject:</b> Re: [cisco-voip] Unity Connection 10.5.2 Split Brain Recovery<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div id=divtagdefaultwrapper><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'>Tom,<o:p></o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><strong><span style='font-family:"Calibri","sans-serif";color:black'>Thin Provisioning:</span></strong><span style='font-family:"Calibri","sans-serif";color:black'><o:p></o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'>Here is a nice article from VMware on thin provisioning: <a href="https://blogs.vmware.com/vsphere/2012/03/thin-provisioning-whats-the-scoop.html" id=LPlnk363941>https://blogs.vmware.com/vsphere/2012/03/thin-provisioning-whats-the-scoop.html</a> Essentially, it is a technique that allows the VM to only consumes storage space needed by the VM, as it needs it up to the size of the storage disk. With respect to Cisco UC, in many cases (DAS TRC) it is not supported and where it is allowed (specs based) you must make sure the space is available at all times (no over-subscription). You can ready more on the subject at <a href="http://docwiki.cisco.com/wiki/UC_Virtualization_Supported_Hardware" title="http://docwiki.cisco.com/wiki/UC_Virtualization_Supported_Hardware
Ctrl+Click or tap to follow the link" id=LPlnk925776>http://docwiki.cisco.com/wiki/UC_Virtualization_Supported_Hardware</a><o:p></o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'>In professional practice I never use thin provisioning for Cisco UC servers.<o:p></o:p></span></p><p class=MsoNormal style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><strong><span style='font-family:"Calibri","sans-serif";color:black'>Unity Connections Split Brain:</span></strong><span style='font-family:"Calibri","sans-serif";color:black'><o:p></o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'>Assuming that you HAD a healthy, well functioning Unity Connections cluster; what typically causes the <em><span style='font-family:"Calibri","sans-serif"'>Split Brain</span></em> issue is when the primary and HA nodes are either rebooted/lose power too close to one another and there are transactions taking place at just the right (rather, wrong) time. Essentially, both nodes end up with different database states and the clustering services cannot accurately determine which database (node) should be primary.<o:p></o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'>You end up in a state where each node is fighting with the other node in an epic "king of the hill" battle. Typically, by taking the HA node off line and rebooting the primary node (and signaling the IVR at least once) the clustering services are able to restore balance to the cluster, then it is usually safe to power the HA node back up.<o:p></o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'>Hope this helps,<o:p></o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'>-Ryan<o:p></o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><p style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'><o:p> </o:p></span></p><div><div class=MsoNormal align=center style='text-align:center;background:white'><span style='font-family:"Calibri","sans-serif";color:black'><hr size=2 width="98%" align=center></span></div><div id=divRplyFwdMsg><p class=MsoNormal style='background:white'><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'> Thomas LeMay <<a href="mailto:thomaslemay@comcast.net">thomaslemay@comcast.net</a>><br><b>Sent:</b> Tuesday, October 27, 2015 2:33 PM<br><b>To:</b> 'Jason Aarons (AM)'; 'Ryan Huff'; 'Aaron Banks'; <a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br><b>Subject:</b> RE: [cisco-voip] Unity Connection 10.5.2 Split Brain Recovery</span><span style='font-family:"Calibri","sans-serif";color:black'> <o:p></o:p></span></p><div><p class=MsoNormal style='background:white'><span style='font-family:"Calibri","sans-serif";color:black'> <o:p></o:p></span></p></div></div><div><div><p style='background:white'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hi, Aaron,</span><span style='color:black'><o:p></o:p></span></p><p style='background:white'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><span style='color:black'><o:p></o:p></span></p><p style='background:white'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Can you elaborate on what is meant by thin provisioning? We experienced split brain a few weeks ago whereby each uc server thought it was the primary. They became locked and I rebooted the servers. When they came back up they went into split brain recovery mode.</span><span style='color:black'><o:p></o:p></span></p><p style='background:white'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><span style='color:black'><o:p></o:p></span></p><p style='background:white'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>We applied the 20K ova template.</span><span style='color:black'><o:p></o:p></span></p><p style='background:white'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><span style='color:black'><o:p></o:p></span></p><p style='background:white'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Tom</span><span style='color:black'><o:p></o:p></span></p><div><div style='border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;border-color:currentColor currentColor;border-image: none'><p style='background:white'><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black'> cisco-voip [<a href="mailto:cisco-voip-bounces@puck.nether.net">mailto:cisco-voip-bounces@puck.nether.net</a>] <b>On Behalf Of </b>Jason Aarons (AM)<br><b>Sent:</b> Tuesday, October 27, 2015 2:17 PM<br><b>To:</b> Ryan Huff; Aaron Banks; <a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br><b>Subject:</b> Re: [cisco-voip] Unity Connection 10.5.2 Split Brain Recovery</span><span style='color:black'><o:p></o:p></span></p></div></div><p style='background:white'><span style='color:black'> <o:p></o:p></span></p><p style='background:white'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Only time I’ve seen split brain in UC was when the customer built the VM with thin provisioning. Seems their vm team did all sorts of things wrong….</span><span style='color:black'><o:p></o:p></span></p><p style='background:white'><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'> </span><span style='color:black'><o:p></o:p></span></p><div><div style='border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;border-color:currentColor currentColor;border-image: none'><p style='background:white'><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:black'> cisco-voip [<a href="mailto:cisco-voip-bounces@puck.nether.net"><span style='color:#0563C1'>mailto:cisco-voip-bounces@puck.nether.net</span></a>] <b>On Behalf Of </b>Ryan Huff<br><b>Sent:</b> Tuesday, October 27, 2015 1:26 PM<br><b>To:</b> Aaron Banks <<a href="mailto:amichaelbanks@hotmail.com"><span style='color:#0563C1'>amichaelbanks@hotmail.com</span></a>>; <a href="mailto:cisco-voip@puck.nether.net"><span style='color:#0563C1'>cisco-voip@puck.nether.net</span></a><br><b>Subject:</b> Re: [cisco-voip] Unity Connection 10.5.2 Split Brain Recovery</span><span style='color:black'><o:p></o:p></span></p></div></div><p style='background:white'><span style='color:black'> <o:p></o:p></span></p><p style='background:white'><span style='color:black'> <o:p></o:p></span></p><div><p style='background:white'><span style='color:black'>1.) Shut down the HA node.<o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'> <o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'>2.) Reboot the primary node<o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'> <o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'>3.) Once the primary node is up, place a call into voicemail <o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'> <o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'>4.) Power the HA node back on<o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'> <o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'>5.) Once HA is up, verify HA status.<o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'> <o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'> <o:p></o:p></span></p></div><div><p style='background:white'><span style='color:black'> <o:p></o:p></span></p></div><div><div><p style='background:white'><span style='font-size:7.0pt;color:#575757'>Sent from my T-Mobile 4G LTE Device</span><span style='color:black'><o:p></o:p></span></p></div></div><p style='background:white'><span style='color:black'><br><br>-------- Original message --------<br>From: Aaron Banks <br>Date:10/27/2015 12:35 PM (GMT-05:00) <br>To: <a href="mailto:cisco-voip@puck.nether.net"><span style='color:#0563C1'>cisco-voip@puck.nether.net</span></a> <br>Subject: [cisco-voip] Unity Connection 10.5.2 Split Brain Recovery <o:p></o:p></span></p><div><p style='background:white'><span style='color:black'><br><br>Has anyone seen/resolved a split brain recovery in Unity Connection 10.5.2? The primary and secondary keep swapping back and forth every few minutes. I can ping and trace to each server. I restarted the primary but that did not resolve the issue. In the RTMT system logs, the secondary sends an NTP query to the primary the response is the primary is inaccessible or down. I'm stumped.<o:p></o:p></span></p></div><p style='background:white'><span style='color:black'><br><br></span><span style='color:white'>itevomcid</span><span style='color:black'> <o:p></o:p></span></p></div></div></div></div></div></body></html>