<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-2022-jp">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>I would start with an NTP service restart first ( save the cluster reboot for after you know it is working
<img style="vertical-align: bottom;" alt="😉" class="EmojiInsert" src="cid:ac73f9d5-978e-4766-8247-cdb66055c019"> )</p>
<p><br>
</p>
<p>Also try the "run sql select name from processnode" and verify the names match reality; improper processnode names will prevent A Cisco DB from starting as well as GUI authentication.<br>
</p>
<p><br>
</p>
<p>Thanks,</p>
<p><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
= Ryan =
<p><br>
</p>
<p><br>
</p>
<p>Email: ryanhuff@outlook.com</p>
<p>Spark: ryanhuff@outlook.com</p>
<p>Twitter: <a id="LPNoLP" href="http://twitter.com/ryanthomashuff">@ryanthomashuff</a><br>
</p>
<p>LinkedIn: <a id="LPNoLP" href="http://linkedin.com/in/ryanthomashuff">ryanthomashuff</a><br>
</p>
<p>Web <a id="LPNoLP" href="http://ryanthomashuff.com">ryanthomashuff.com</a></p>
</div>
</div>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Andy Carse <andy.carse@gmail.com><br>
<b>Sent:</b> Wednesday, March 2, 2016 2:27 PM<br>
<b>To:</b> Lelio Fulgenzi<br>
<b>Cc:</b> Ryan Huff; Cisco VoIP List<br>
<b>Subject:</b> Re: [cisco-voip] CUCM Upgrade woes</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div>
<div>Yes I spotted that after I pasted,<br>
</div>
I'm not sure how it installed v9.1 with that stratum.<br>
</div>
I've changed it now to be a 3 but I guess that its too late.<br>
</div>
I'll try a reboot and see if that fixes it <br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 2 March 2016 at 19:23, Lelio Fulgenzi <span dir="ltr">
<<a href="mailto:lelio@uoguelph.ca" target="_blank">lelio@uoguelph.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div>
<div style="font-family:Verdana; font-size:10pt; color:#000000">
<div><br>
</div>
<div>I could be wrong, but it looks like your NTP is not synchronizing properly. It may not be the issue, but it certainly doesn't help.</div>
<div><br>
</div>
<div><br>
</div>
<div>Here's a sample of ours with what I think it should look like.</div>
<div><br>
</div>
<div>
<hr>
<p style="margin:0px"><span style="font-family:'lucida console',sans-serif">ntpd (pid 6691) is running...</span></p>
<p style="margin:0px"><span class=""><span style="font-family:'lucida console',sans-serif">remote refid st t when poll reach delay offset jitter</span><br>
<span style="font-family:'lucida console',sans-serif">==============================================================================</span><br>
</span><span style="font-family:'lucida console',sans-serif">127.127.1.0 .LOCL. 10 l 2 64 377 0.000 0.000 0.001</span><br>
<span style="font-family:'lucida console',sans-serif">*xxx.xxx.xxx.201 xxx.xxx.xxx.53 3 u 721 1024 377 0.663 -0.256 0.124</span><br>
<span style="font-family:'lucida console',sans-serif">+xxx.xxx.xxx.201 xxx.xxx.xxx.53 3 u 1021 1024 377 0.924 0.528 0.014</span></p>
<p style="margin:0px"><br>
<span style="font-family:'lucida console',sans-serif">synchronised to NTP server (xxx.xxx.xxx.201) at stratum 4</span><br>
<span style="font-family:'lucida console',sans-serif">time correct to within 86 ms</span><br>
<span style="font-family:'lucida console',sans-serif">polling server every 1024 s</span></p>
<hr>
<p style="margin:0px"><br>
</p>
<p style="margin:0px"><br>
</p>
</div>
<div><span name="x"></span>---<br>
Lelio Fulgenzi, B.A.<br>
Senior Analyst, Network Infrastructure<br>
Computing and Communications Services (CCS)<br>
University of Guelph<br>
<div><br>
</div>
519$B!>(B824$B!>(B4120 Ext 56354<br>
<a href="mailto:lelio@uoguelph.ca" target="_blank">lelio@uoguelph.ca</a><br>
<a href="http://www.uoguelph.ca/ccs" target="_blank">www.uoguelph.ca/ccs</a><br>
Room 037, Animal Science and Nutrition Building<br>
Guelph, Ontario, N1G 2W1<span name="x"></span><br>
</div>
<div><br>
</div>
<hr>
<div style="color:#000; font-weight:normal; font-style:normal; text-decoration:none; font-family:Helvetica,Arial,sans-serif; font-size:12pt">
<b>From: </b>"Andy Carse" <<a href="mailto:andy.carse@gmail.com" target="_blank">andy.carse@gmail.com</a>><br>
<b>To: </b>"Ryan Huff" <<a href="mailto:ryanhuff@outlook.com" target="_blank">ryanhuff@outlook.com</a>><span class=""><br>
<b>Cc: </b>"Cisco VoIP List" <<a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a>><br>
</span><b>Sent: </b>Wednesday, March 2, 2016 2:10:31 PM<span class=""><br>
<b>Subject: </b>Re: [cisco-voip] CUCM Upgrade woes<br>
<div><br>
</div>
</span>
<div>
<div class="h5">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>.The upgrade was from 9.1.2 to 10.5.2.13900-12.<br>
</div>
</div>
There where some issues with having the GBNP installed so a direct upgrade was a non-starter.<br>
</div>
This is a hardware refresh and software upgrade rolled up into one project.<br>
<div><br>
</div>
</div>
So I backed up the 9.1 <br>
</div>
installed 9.1 on the new hardware to the same specifications, IP addresses and OVA etc<br>
</div>
Restored on to the new publisher ok.<br>
</div>
installed the cop files as required, then had fun with the GBNP.<br>
</div>
So I exported everything except route patterns from the production system.<br>
</div>
Rebuilt another 9.1 this time not installing GBNP.<br>
</div>
Imported into this new cluster.<br>
</div>
Upgraded to 10.5.2.10000-5.<br>
</div>
Then upgraded to 10.5.13900-12.<br>
</div>
Didn't seem to be an issue then came in today and its broken.<br>
<div><br>
</div>
</div>
The info you requested is pasted below<br>
<div><br>
</div>
admin:file view activelog platform/log/diag1.log<br>
<div><br>
</div>
03-02-2016 18:46:31 Diagnostics Version: 1.0.0<br>
03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel]<br>
03-02-2016 18:46:32 Hardware Model: VMware<br>
03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"]<br>
03-02-2016 18:46:32 Version: 10.5.2<br>
03-02-2016 18:46:33 disk_space: Is valid module: True<br>
03-02-2016 18:46:33 disk_files: Is valid module: True<br>
03-02-2016 18:46:33 service_manager: Is valid module: True<br>
03-02-2016 18:46:33 tomcat: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_keystore: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_connectors: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_threads: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_memory: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_sessions: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True<br>
03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml<br>
03-02-2016 18:46:33 validate_network: val: true<br>
03-02-2016 18:46:33 validate_network: Is valid module: True<br>
03-02-2016 18:46:33 validate_network_adv: Is valid module: False<br>
<div><br>
</div>
options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : <br>
<div><br>
</div>
<br>
admin:<br>
admin:file view activelog platform/log/diag1.log<br>
<div><br>
</div>
03-02-2016 18:46:31 Diagnostics Version: 1.0.0<br>
03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel]<br>
03-02-2016 18:46:32 Hardware Model: VMware<br>
03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"]<br>
03-02-2016 18:46:32 Version: 10.5.2<br>
03-02-2016 18:46:33 disk_space: Is valid module: True<br>
03-02-2016 18:46:33 disk_files: Is valid module: True<br>
03-02-2016 18:46:33 service_manager: Is valid module: True<br>
03-02-2016 18:46:33 tomcat: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_keystore: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_connectors: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_threads: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_memory: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_sessions: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True<br>
03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml<br>
03-02-2016 18:46:33 validate_network: val: true<br>
03-02-2016 18:46:33 validate_network: Is valid module: True<br>
03-02-2016 18:46:33 validate_network_adv: Is valid module: False<br>
<div><br>
</div>
options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : <br>
03-02-2016 18:46:33 raid: getting cpu speed [/usr/local/bin/base_scripts/sd_hwdetect CPUSpeed]<br>
03-02-2016 18:46:33 raid: CPU Speed: 2500<br>
03-02-2016 18:46:33 raid: model = VMware<br>
03-02-2016 18:46:33 raid: Is valid module: True<br>
03-02-2016 18:46:33 system_info: Is valid module: True<br>
03-02-2016 18:46:33 ntp_reachability: Is valid module: True<br>
03-02-2016 18:46:33 ntp_clock_drift: Is valid module: True<br>
03-02-2016 18:46:33 ntp_stratum: Is valid module: True<br>
03-02-2016 18:46:33 sdl_fragmentation: Is valid module: True<br>
03-02-2016 18:46:33 sdi_fragmentation: Is valid module: True<br>
03-02-2016 18:46:33 ipv6_networking: IPV6INIT=no<br>
03-02-2016 18:46:33 ipv6_networking: IPv6 initialized: no<br>
03-02-2016 18:46:33 ipv6_networking: False<br>
03-02-2016 18:46:33 ipv6_networking: Is valid module: False<br>
03-02-2016 18:46:33 <br>
03-02-2016 18:46:33 --> executing test [validate_network], fix: fixauto, stop on error: False<br>
03-02-2016 18:46:33 <br>
03-02-2016 18:46:33 validate_network: ------------------<br>
03-02-2016 18:46:33 validate_network: Testing networking, but skipping duplicate IP test.<br>
03-02-2016 18:46:33 validate_network: checking network [/usr/local/bin/base_scripts/validateNetworking.sh -n]<br>
<div><br>
</div>
options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 54) : <br>
03-02-2016 18:46:33 validate_network: retrieving pub name from [/usr/local/platform/conf/platformConfig.xml]<br>
03-02-2016 18:46:33 validate_network: Hostname: [XXXXXXXX]<br>
03-02-2016 18:46:33 validate_network: found pub name [XXXXXXX]<br>
03-02-2016 18:46:33 validate_network: checking /etc/hosts [grep -q `hostname` /etc/hosts]<br>
03-02-2016 18:46:33 validate_network: Finding cluster nodes [/usr/local/bin/base_scripts/list_cluster.sh]<br>
03-02-2016 18:46:33 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.101>/dev/null]<br>
03-02-2016 18:46:35 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.102>/dev/null]<br>
03-02-2016 18:46:46 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.4x.101>/dev/null]<br>
03-02-2016 18:46:47 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.130>/dev/null]<br>
03-02-2016 18:46:48 validate_network: does test script exist [/usr/local/bin/base_scripts/networkDiagnostic.sh]<br>
03-02-2016 18:46:48 validate_network: test script exists<br>
03-02-2016 18:46:48 validate_network: run network script via expect [./diag_validate_network.exp > /dev/null]<br>
03-02-2016 18:46:48 validate_network: result: 0, message: Passed <br>
<div><br>
</div>
<br>
end of the file reached<br>
options: q=quit, n=next, p=prev, b=begin, e=end (lines 41 - 54 of 54) : <br>
admin:<br>
<div><br>
</div>
<br>
<div><br>
</div>
admin:utils ntp status<br>
ntpd (pid 8970) is running...<br>
<div><br>
</div>
remote refid st t when poll reach delay offset jitter<br>
==============================================================================<br>
xxx.xxx.55.203 .INIT. 16 u - 1024 0 0.000 0.000 0.000<br>
*xxx.xxx.5.203 LOCAL(1) 8 u 268 512 377 0.611 0.304 0.289<br>
<div><br>
</div>
<br>
synchronised to NTP server (xxx.xxx.5.203) at stratum 9 <br>
time correct to within 37 ms<br>
polling server every 512 s<br>
<div><br>
</div>
Current time in UTC is : Wed Mar 2 18:49:09 UTC 2016<br>
Current time in Europe/London is : Wed Mar 2 18:49:09 GMT 2016<br>
admin:<br>
<div><br>
</div>
<br>
<div><br>
</div>
admin:utils ntp server list<br>
xxx.xxx.55.203<br>
<div><br>
</div>
xxx.xxx.5.203<br>
admin:<br>
<div><br>
</div>
<br>
</div>
Regards <br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 2 March 2016 at 17:36, Ryan Huff <span dir="ltr"><<a href="mailto:ryanhuff@outlook.com" target="_blank">ryanhuff@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
Not that I'm suggesting you not call TAC but the engineer in me just keeps going ....<br>
<br>
What (version) did you upgrade from and did you upgrade in-place VMs, DRS/Rebuild or P->V?<br>
<br>
Do you know if at any point post upgrade, the cluster was healthy and then failed or it has always been in a degraded state since the upgrade?<br>
<br>
Can you show me the output (from the publisher);<br>
<br>
- utils diagnose module validate_network<br>
- show ntp status<br>
- show ntp server list<br>
<br>
Thanks,<br>
<br>
Ryan<br>
<div>
<div><br>
> On Mar 2, 2016, at 12:25 PM, Ryan Huff <<a href="mailto:ryanhuff@outlook.com" target="_blank">ryanhuff@outlook.com</a>> wrote:<br>
><br>
> I'd go through a quick checklist while calling in a severity 1 TAC case;<br>
><br>
> - forward and reverse DNS for all cluster nodes (and resolving to the correct addresses)<br>
><br>
> - verify the processNodes, if using hosts or fqdn, are correctly resolvable. This will prevent A Cisco DB from starting as well as GUI authentication<br>
><br>
> - do not have an absurd clock sync on the nodes (Stratum 3 or better)<br>
><br>
> Thanks,<br>
><br>
> Ryan<br>
><br>
>> On Mar 2, 2016, at 12:13 PM, Andy Carse <<a href="mailto:andy.carse@gmail.com" target="_blank">andy.carse@gmail.com</a>> wrote:<br>
>><br>
>> I thought I was home and dry with this upgrade, but it would seem that the gods have deserted me.<br>
>><br>
>> I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything seemed ok.<br>
>> This morning I've come in to find that the database on the publisher won't start.<br>
>> So I've tried<br>
>> 1. reboot of the cluster (its not gone live yet) no change.<br>
>> 2. Utils service start A Cisco DB<br>
>> 2. tried dbreplication stop on the subs, then the publisher.<br>
>> dbreplication dropddmindb on the subs<br>
>> dbreplication dropadmindb on the pub<br>
>> The pub comes back with "DropAdminDB cannot be executed on standalone or Cores cluster"<br>
>><br>
>> I can't even web to ccmadmin on the pub and I forgot to carry out the "Golden Rule" of taking a backup soon after the upgrade.<br>
>> If I try to RTM that also fails......<br>
>><br>
>> Is it time for a start from scratch moment?<br>
>><br>
>><br>
>><br>
>> --<br>
>> Rgds Andy<br>
>><br>
>> _______________________________________________<br>
>> cisco-voip mailing list<br>
>> <a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
>> <a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">
https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
> _______________________________________________<br>
> cisco-voip mailing list<br>
> <a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">
https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div>Rgds Andy<br>
<div><br>
</div>
</div>
</div>
<br>
_______________________________________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
</div>
</div>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">Rgds Andy<br>
<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>