<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#404040;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle19
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Just a quick wrap-up on this one…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Two defects created for this problem are CSCup27726 and CSCup27133.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">- Dan<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#404040"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Wes Sisk (wsisk) [mailto:wsisk@cisco.com]
<br>
<b>Sent:</b> Wednesday, May 21, 2014 2:50 PM<br>
<b>To:</b> Daniel Pagan<br>
<b>Cc:</b> cisco-voip@puck.nether.net<br>
<b>Subject:</b> Re: [cisco-voip] Heartbeat Failure & SNRD<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Daniel, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Great find!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">For the document:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="http://www.cisco.com/c/en/us/support/docs/voice-unified-communications/unified-communications-manager-callmanager/46806-cm-crashes-and-shutdowns.html">http://www.cisco.com/c/en/us/support/docs/voice-unified-communications/unified-communications-manager-callmanager/46806-cm-crashes-and-shutdowns.html</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The initialization process and timers have changed *significantly* since 4.x. Some examples include:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">CSCsj76788 cp-system request to remove initialization timers<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">“... remove the initialization timers that are started during CUCM initialization. These timer would previously cause a system restart under certain circumstance…”<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Still, there is a global maximum timeout. Individual Daemons must report start and successful initiation by that time.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Historically behavior like you discuss was triggered by service parameters being missing or having incorrect values. This may be a problem with connection to the database ( CSCsc72748 ) or problem with the contents of the database. Other
problems include another process grabbing one of the TCP or UDP ports required by the ccm process.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">ccm had many issues retrieving initialization information from the database in early linux versions. refinements to informix and in memory database (IMDB) have helped significantly.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-Wes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On May 21, 2014, at 9:33 AM, Daniel Pagan <<a href="mailto:dpagan@fidelus.com">dpagan@fidelus.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Folks:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">CUCM ES 8.6.2.24122-1 appears to be creating an issue where CallManager heartbeat fails to increment upon startup and the condition that must be met is very specific. On
a problematic node, SDL traces show the following error exactly one hour after the start of the CCM service:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#C00000">AppError ||||||Local send blocked: SignalName: Start, DestPID:<span class="apple-converted-space"> </span><span style="background:yellow">SNRD</span>[1:100:61:1]</span></i></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#C00000"> </span></i></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">This error is followed by the SDL trace printing an error stating CallManager exceeded the permitted time for initialization and will restart the application. The CCM application
restarts and additional SDL traces are printed showing the standard creation of critical processes – one hour later the same “Local send blocked” error is printed regarding the SNRD process.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">I saw the<span class="apple-converted-space"> </span><b>DestPID: SNRD</b><span class="apple-converted-space"> </span>error, went to a completely different,<span class="apple-converted-space"> </span><b>non-problematic</b><span class="apple-converted-space"> </span>lab
environment where 8.6.2.24122-1 is installed, created a single Remote Destination Profile, and then restarted the standalone node in order to force the creation of SNRD. CallManager heartbeats are now failing to increment in that environment and found another
“Local send blocked” error regarding SNRD. Removing the single Remote Destination Profile from the standalone environment and rebooting the node resolves the problem. Re-inserting it again followed by a reboot recreates it, making SNRD the obvious culprit
here.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">I currently have a TAC case open where they’re attempting to recreate the problem. It seems no public facing defects are created for this.<span style="color:#1F497D"> </span>Just
wanted to give you folks a heads up.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Related to this, can someone tell me if this document, specifally the section describing MMManInit and process creation, is still accurate? If so, then what I fail to see
in SDL traces is a<span class="apple-converted-space"> </span><b>InitDone</b><span class="apple-converted-space"> </span>signal from SNRD to MMManInit during the 60 minutes between CCM startup and initialization timeout.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">- Daniel<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> <o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif"">_______________________________________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net"><span style="color:#954F72">cisco-voip@puck.nether.net</span></a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip"><span style="color:#954F72">https://puck.nether.net/mailman/listinfo/cisco-voip</span></a><o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>