<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
Andy,<br>
<br>
<p>Sorry for the late relpy .. the mailing list seems delayed. I noticed the your clock sync is stratum 9 .... that is super high. While 37 milliseconds may seem trivial in the grand scheme of things, high clock sync in UC servers (> 3 hops) WILL cause some
of the strangest, most seemingly unrelated and crazy behavior.</p>
<p><br>
</p>
<p>The SRND recommends Strata 3 or better; I personally say strata 2 or better. Get your clock sync down, restart NTP service and possibly attempt to reset the cluster repliction and/or a cluster reboot</p>
<p>Also, from the CLI, run <i>run sql select name from processnode</i> Do the hostnames / FQDNs returned match up with reality?<br>
</p>
<p><br>
</p>
<p><br>
</p>
<div id="Signature">
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
= Ryan =
<p><br>
</p>
<p><br>
</p>
<p>Email: ryanhuff@outlook.com</p>
<p>Spark: ryanhuff@outlook.com</p>
<p>Twitter: <a id="LPNoLP" href="http://twitter.com/ryanthomashuff">@ryanthomashuff</a><br>
</p>
<p>LinkedIn: <a id="LPNoLP" href="http://linkedin.com/in/ryanthomashuff">ryanthomashuff</a><br>
</p>
<p>Web <a id="LPNoLP" href="http://ryanthomashuff.com">ryanthomashuff.com</a></p>
</div>
</div>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Andy Carse <andy.carse@gmail.com><br>
<b>Sent:</b> Wednesday, March 2, 2016 2:10 PM<br>
<b>To:</b> Ryan Huff<br>
<b>Cc:</b> Cisco VoIP List<br>
<b>Subject:</b> Re: [cisco-voip] CUCM Upgrade woes</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>.The upgrade was from 9.1.2 to 10.5.2.13900-12.<br>
</div>
</div>
There where some issues with having the GBNP installed so a direct upgrade was a non-starter.<br>
</div>
This is a hardware refresh and software upgrade rolled up into one project.<br>
<br>
</div>
So I backed up the 9.1 <br>
</div>
installed 9.1 on the new hardware to the same specifications, IP addresses and OVA etc<br>
</div>
Restored on to the new publisher ok.<br>
</div>
installed the cop files as required, then had fun with the GBNP.<br>
</div>
So I exported everything except route patterns from the production system.<br>
</div>
Rebuilt another 9.1 this time not installing GBNP.<br>
</div>
Imported into this new cluster.<br>
</div>
Upgraded to 10.5.2.10000-5.<br>
</div>
Then upgraded to 10.5.13900-12.<br>
</div>
Didn't seem to be an issue then came in today and its broken.<br>
<br>
</div>
The info you requested is pasted below<br>
<br>
admin:file view activelog platform/log/diag1.log<br>
<br>
03-02-2016 18:46:31 Diagnostics Version: 1.0.0<br>
03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel]<br>
03-02-2016 18:46:32 Hardware Model: VMware<br>
03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"]<br>
03-02-2016 18:46:32 Version: 10.5.2<br>
03-02-2016 18:46:33 disk_space: Is valid module: True<br>
03-02-2016 18:46:33 disk_files: Is valid module: True<br>
03-02-2016 18:46:33 service_manager: Is valid module: True<br>
03-02-2016 18:46:33 tomcat: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_keystore: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_connectors: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_threads: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_memory: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_sessions: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True<br>
03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml<br>
03-02-2016 18:46:33 validate_network: val: true<br>
03-02-2016 18:46:33 validate_network: Is valid module: True<br>
03-02-2016 18:46:33 validate_network_adv: Is valid module: False<br>
<br>
options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : <br>
<br>
<br>
admin:<br>
admin:file view activelog platform/log/diag1.log<br>
<br>
03-02-2016 18:46:31 Diagnostics Version: 1.0.0<br>
03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel]<br>
03-02-2016 18:46:32 Hardware Model: VMware<br>
03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"]<br>
03-02-2016 18:46:32 Version: 10.5.2<br>
03-02-2016 18:46:33 disk_space: Is valid module: True<br>
03-02-2016 18:46:33 disk_files: Is valid module: True<br>
03-02-2016 18:46:33 service_manager: Is valid module: True<br>
03-02-2016 18:46:33 tomcat: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_keystore: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_connectors: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_threads: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_memory: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_sessions: Is valid module: True<br>
03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True<br>
03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml<br>
03-02-2016 18:46:33 validate_network: val: true<br>
03-02-2016 18:46:33 validate_network: Is valid module: True<br>
03-02-2016 18:46:33 validate_network_adv: Is valid module: False<br>
<br>
options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : <br>
03-02-2016 18:46:33 raid: getting cpu speed [/usr/local/bin/base_scripts/sd_hwdetect CPUSpeed]<br>
03-02-2016 18:46:33 raid: CPU Speed: 2500<br>
03-02-2016 18:46:33 raid: model = VMware<br>
03-02-2016 18:46:33 raid: Is valid module: True<br>
03-02-2016 18:46:33 system_info: Is valid module: True<br>
03-02-2016 18:46:33 ntp_reachability: Is valid module: True<br>
03-02-2016 18:46:33 ntp_clock_drift: Is valid module: True<br>
03-02-2016 18:46:33 ntp_stratum: Is valid module: True<br>
03-02-2016 18:46:33 sdl_fragmentation: Is valid module: True<br>
03-02-2016 18:46:33 sdi_fragmentation: Is valid module: True<br>
03-02-2016 18:46:33 ipv6_networking: IPV6INIT=no<br>
03-02-2016 18:46:33 ipv6_networking: IPv6 initialized: no<br>
03-02-2016 18:46:33 ipv6_networking: False<br>
03-02-2016 18:46:33 ipv6_networking: Is valid module: False<br>
03-02-2016 18:46:33 <br>
03-02-2016 18:46:33 --> executing test [validate_network], fix: fixauto, stop on error: False<br>
03-02-2016 18:46:33 <br>
03-02-2016 18:46:33 validate_network: ------------------<br>
03-02-2016 18:46:33 validate_network: Testing networking, but skipping duplicate IP test.<br>
03-02-2016 18:46:33 validate_network: checking network [/usr/local/bin/base_scripts/validateNetworking.sh -n]<br>
<br>
options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 54) : <br>
03-02-2016 18:46:33 validate_network: retrieving pub name from [/usr/local/platform/conf/platformConfig.xml]<br>
03-02-2016 18:46:33 validate_network: Hostname: [XXXXXXXX]<br>
03-02-2016 18:46:33 validate_network: found pub name [XXXXXXX]<br>
03-02-2016 18:46:33 validate_network: checking /etc/hosts [grep -q `hostname` /etc/hosts]<br>
03-02-2016 18:46:33 validate_network: Finding cluster nodes [/usr/local/bin/base_scripts/list_cluster.sh]<br>
03-02-2016 18:46:33 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.101>/dev/null]<br>
03-02-2016 18:46:35 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.102>/dev/null]<br>
03-02-2016 18:46:46 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.4x.101>/dev/null]<br>
03-02-2016 18:46:47 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.130>/dev/null]<br>
03-02-2016 18:46:48 validate_network: does test script exist [/usr/local/bin/base_scripts/networkDiagnostic.sh]<br>
03-02-2016 18:46:48 validate_network: test script exists<br>
03-02-2016 18:46:48 validate_network: run network script via expect [./diag_validate_network.exp > /dev/null]<br>
03-02-2016 18:46:48 validate_network: result: 0, message: Passed <br>
<br>
<br>
end of the file reached<br>
options: q=quit, n=next, p=prev, b=begin, e=end (lines 41 - 54 of 54) : <br>
admin:<br>
<br>
<br>
<br>
admin:utils ntp status<br>
ntpd (pid 8970) is running...<br>
<br>
remote refid st t when poll reach delay offset jitter<br>
==============================================================================<br>
xxx.xxx.55.203 .INIT. 16 u - 1024 0 0.000 0.000 0.000<br>
*xxx.xxx.5.203 LOCAL(1) 8 u 268 512 377 0.611 0.304 0.289<br>
<br>
<br>
synchronised to NTP server (xxx.xxx.5.203) at stratum 9 <br>
time correct to within 37 ms<br>
polling server every 512 s<br>
<br>
Current time in UTC is : Wed Mar 2 18:49:09 UTC 2016<br>
Current time in Europe/London is : Wed Mar 2 18:49:09 GMT 2016<br>
admin:<br>
<br>
<br>
<br>
admin:utils ntp server list<br>
xxx.xxx.55.203<br>
<br>
xxx.xxx.5.203<br>
admin:<br>
<br>
<br>
</div>
Regards <br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 2 March 2016 at 17:36, Ryan Huff <span dir="ltr"><<a href="mailto:ryanhuff@outlook.com" target="_blank">ryanhuff@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
Not that I'm suggesting you not call TAC but the engineer in me just keeps going ....<br>
<br>
What (version) did you upgrade from and did you upgrade in-place VMs, DRS/Rebuild or P->V?<br>
<br>
Do you know if at any point post upgrade, the cluster was healthy and then failed or it has always been in a degraded state since the upgrade?<br>
<br>
Can you show me the output (from the publisher);<br>
<br>
- utils diagnose module validate_network<br>
- show ntp status<br>
- show ntp server list<br>
<br>
Thanks,<br>
<br>
Ryan<br>
<div class="HOEnZb">
<div class="h5"><br>
> On Mar 2, 2016, at 12:25 PM, Ryan Huff <<a href="mailto:ryanhuff@outlook.com">ryanhuff@outlook.com</a>> wrote:<br>
><br>
> I'd go through a quick checklist while calling in a severity 1 TAC case;<br>
><br>
> - forward and reverse DNS for all cluster nodes (and resolving to the correct addresses)<br>
><br>
> - verify the processNodes, if using hosts or fqdn, are correctly resolvable. This will prevent A Cisco DB from starting as well as GUI authentication<br>
><br>
> - do not have an absurd clock sync on the nodes (Stratum 3 or better)<br>
><br>
> Thanks,<br>
><br>
> Ryan<br>
><br>
>> On Mar 2, 2016, at 12:13 PM, Andy Carse <<a href="mailto:andy.carse@gmail.com">andy.carse@gmail.com</a>> wrote:<br>
>><br>
>> I thought I was home and dry with this upgrade, but it would seem that the gods have deserted me.<br>
>><br>
>> I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything seemed ok.<br>
>> This morning I've come in to find that the database on the publisher won't start.<br>
>> So I've tried<br>
>> 1. reboot of the cluster (its not gone live yet) no change.<br>
>> 2. Utils service start A Cisco DB<br>
>> 2. tried dbreplication stop on the subs, then the publisher.<br>
>> dbreplication dropddmindb on the subs<br>
>> dbreplication dropadmindb on the pub<br>
>> The pub comes back with "DropAdminDB cannot be executed on standalone or Cores cluster"<br>
>><br>
>> I can't even web to ccmadmin on the pub and I forgot to carry out the "Golden Rule" of taking a backup soon after the upgrade.<br>
>> If I try to RTM that also fails......<br>
>><br>
>> Is it time for a start from scratch moment?<br>
>><br>
>><br>
>><br>
>> --<br>
>> Rgds Andy<br>
>><br>
>> _______________________________________________<br>
>> cisco-voip mailing list<br>
>> <a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
>> <a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">
https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
> _______________________________________________<br>
> cisco-voip mailing list<br>
> <a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">
https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">Rgds Andy<br>
<br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>