<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Arial","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Arial","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="2050" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>I was fortunate enough to be greeted by the following errors
in the event log on both servers last night after simply rebooting our 4.1.3
cluster:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Event Type: Error<o:p></o:p></p>
<p class=MsoNormal>Event Source: Cisco CallManager<o:p></o:p></p>
<p class=MsoNormal>Event Category: None<o:p></o:p></p>
<p class=MsoNormal>Event ID: 3<o:p></o:p></p>
<p class=MsoNormal>Date: 8/14/2008<o:p></o:p></p>
<p class=MsoNormal>Time: 10:57:31
PM<o:p></o:p></p>
<p class=MsoNormal>User: N/A<o:p></o:p></p>
<p class=MsoNormal>Computer: HOUCM01<o:p></o:p></p>
<p class=MsoNormal>Description:<o:p></o:p></p>
<p class=MsoNormal>Error: DeviceInitTimerPop - Device Initialization timer has
expired.<o:p></o:p></p>
<p class=MsoNormal> Timer Value in seconds: 360<o:p></o:p></p>
<p class=MsoNormal> App ID: Cisco CallManager<o:p></o:p></p>
<p class=MsoNormal> Cluster ID: US-HOU-CL01<o:p></o:p></p>
<p class=MsoNormal> Node ID: x.x.x.x<o:p></o:p></p>
<p class=MsoNormal>Explanation: The Device Initialization process has taken
longer than expected to initialize. The system configuration is larger
and/or more complex than expected, or an error has been encountered.<o:p></o:p></p>
<p class=MsoNormal>Recommended Action: Try increasing the "Device
Initialization Timer" value in the CallManager Service Parameters
administration page in 50% multiples of the suggested default value. <o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>*Note: This parameter is cluster-wide. Setting this
parameter will effect each call processing server within the cluster on the
next service restart.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>If the system does not successfully initialize by increasing
this parameter parameter to the maximum value, contact Cisco TAC support
immediately..<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<div style='mso-element:para-border-div;border:none;border-bottom:dotted windowtext 3.0pt;
padding:0in 0in 1.0pt 0in'>
<p class=MsoNormal style='border:none;padding:0in'><o:p> </o:p></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>And this gem of an error:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Event Type: Error<o:p></o:p></p>
<p class=MsoNormal>Event Source: Cisco CallManager<o:p></o:p></p>
<p class=MsoNormal>Event Category: None<o:p></o:p></p>
<p class=MsoNormal>Event ID: 3<o:p></o:p></p>
<p class=MsoNormal>Date: 8/14/2008<o:p></o:p></p>
<p class=MsoNormal>Time: 10:57:31
PM<o:p></o:p></p>
<p class=MsoNormal>User: N/A<o:p></o:p></p>
<p class=MsoNormal>Computer: HOUCM01<o:p></o:p></p>
<p class=MsoNormal>Description:<o:p></o:p></p>
<p class=MsoNormal>Error: CallManagerFailure - Indicates some failure in the
Cisco CallManager system.<o:p></o:p></p>
<p class=MsoNormal> Host name of hosting node.: HOUCM01<o:p></o:p></p>
<p class=MsoNormal> IP address of hosting node.: x.x.x.x<o:p></o:p></p>
<p class=MsoNormal> Reason code.: 5<o:p></o:p></p>
<p class=MsoNormal> Additional Text [Optional]:
MMManInit::initializing_devices<o:p></o:p></p>
<p class=MsoNormal> App ID: Cisco CallManager<o:p></o:p></p>
<p class=MsoNormal> Cluster ID: US-HOU-CL01<o:p></o:p></p>
<p class=MsoNormal> Node ID: x.x.x.x<o:p></o:p></p>
<p class=MsoNormal>Explanation: This alarm indicates that some failure occurred
in the Cisco CallManager system.<o:p></o:p></p>
<p class=MsoNormal>Recommended Action: Monitor for other alarms and restart
Cisco CallManager service, if necessary..<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<div style='mso-element:para-border-div;border:none;border-bottom:dotted windowtext 3.0pt;
padding:0in 0in 1.0pt 0in'>
<p class=MsoNormal style='border:none;padding:0in'><o:p> </o:p></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>The CallManager service kept ‘terminating unexpectedly’
every 10 minutes or so on both servers and therefore not a single device in the
entire cluster would register to either the pub or the sub. I ended up
calling TAC since I had never seen these two errors before and tried
increasing the Device Initialization Timer, but it did not help. Almost
2 hours into it I realized that when I updated this service parameter it did
not get changed for some reason when I did it earlier and I remember TAC seeing
the service parameter page with a bunch of errors on it. So I thought I had
updated it, but it errored out and I didn’t realize it. So I upped
it from 360 to 550, restarted CallManager and DBL services on both servers and
5 minutes later all the phones registered and peace was restored.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>TAC has no answer for what may have happened other than
there was a Dr. Watson log generated right when I rebooted the pub which they
are analyzing. So ultimately my question is why would the ‘device
subsystem’ suddenly not initialize within the default 360 seconds? What
exactly is the device subsystem other than what the name implies? I can
only assume changing this value to 550 is what restored order in the
cluster. There are less than 1800 phones in the database and just under
800 gateways and probably a quarter of them are old phones and gateways that
were never deleted. This is the HP equivalent MSC7835 bought last year so
the hardware is current. I wouldn’t think there are so many
devices to initialize that this timer had to be upped. Or is it? Has
anyone ever had to change this timer?<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>I’m at a loss and have to explain why the entire US
cluster including 9 6608 PRI’s was down hard for almost 3 hours after a
run of the mill preventative maintenance quarterly reboot.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Thanks for any input!<o:p></o:p></p>
<p class=MsoNormal><b><span style='color:black'><o:p> </o:p></span></b></p>
<p class=MsoNormal><b><span style='color:black'>....................................................................................
</span></b><span style='color:black'><o:p></o:p></span></p>
<p class=MsoNormal><b><span style='font-size:11.0pt;color:gray'><o:p> </o:p></span></b></p>
<p class=MsoNormal><b><span style='color:gray'>Keith Klevenski<o:p></o:p></span></b></p>
<p class=MsoNormal><b><span style='color:gray'>Manager, Telephony Engineering<o:p></o:p></span></b></p>
<p class=MsoNormal><span style='font-size:12.0pt;color:navy'><o:p> </o:p></span></p>
<p class=MsoNormal><b><span style='font-size:9.0pt;color:gray'>1880 S. Dairy
Ashford, Suite 300 </span></b><b><span style='font-size:9.0pt;color:#FF9900'>|
</span></b><b><span style='font-size:9.0pt;color:gray'> Houston, TX
77077- 4760</span></b><b><span style='font-size:11.0pt;color:black'><o:p></o:p></span></b></p>
<p class=MsoNormal><span style='font-size:4.0pt;color:gray'><br>
</span><b><span lang=NO-BOK style='font-size:9.0pt;color:gray'>Tel:
281.674.0702 </span></b><b><span lang=NO-BOK style='font-size:9.0pt;
color:#FF9900'>| </span></b><b><span lang=NO-BOK style='font-size:9.0pt;
color:gray'> Mobile: 713.677.3925 </span></b><b><span lang=NO-BOK
style='font-size:9.0pt;color:#FF9900'>| </span></b><b><span
lang=NO-BOK style='font-size:9.0pt;color:gray'>Fax: 281.674.0101<o:p></o:p></span></b></p>
<p class=MsoNormal style='mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:
2.0pt;margin-left:0in'><a name="OLE_LINK2"></a><a name="OLE_LINK1"></a><span
style='font-size:11.0pt'><a href="mailto:keith.klevenski@rig.net"><span
lang=NO-BOK style='font-size:9.0pt;color:blue'>keith.klevenski@rig.net</span></a><span
style='color:black'><o:p></o:p></span></span></p>
<p class=MsoNormal style='mso-margin-top-alt:2.0pt;margin-right:0in;margin-bottom:
2.0pt;margin-left:0in'><span style='font-size:11.0pt;color:black'><a
href="http://www.rig.net/"><b><span lang=NO-BOK style='font-size:9.0pt;
color:#FF9900'>www.rig.net</span></b></a><o:p></o:p></span></p>
<p class=MsoNormal style='mso-margin-top-alt:2.0pt;margin-right:0in;margin-bottom:
2.0pt;margin-left:0in'><span style='font-size:12.0pt;font-family:"Times New Roman","serif";
color:black'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:navy'><img border=0 width=135 height=43 id="Picture_x0020_1"
src="cid:image001.jpg@01C8FEC5.E38E8600"
alt="cid:image001.jpg@01C890BC.B777EB00"></span><span style='font-size:11.0pt;
font-family:"Calibri","sans-serif";color:navy'><o:p></o:p></span></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>