<html><body><div style="font-family: Verdana; font-size: 10pt; color: #000000"><div><br></div><div>i'd first try getting NTP to work first on all the hosts.</div><div><br></div><div>a reboot might fix things, but it might not. sometimes, the database sync needs to be restarted manually and won't restart even after a restart of the cluster.</div><div><br></div><div>i'd engage the TAC after the NTP is fixed.</div><div><br></div><div><br></div><div><span name="x"></span>---<br>Lelio Fulgenzi, B.A.<br>Senior Analyst, Network Infrastructure<br>Computing and Communications Services (CCS)<br>University of Guelph<br><div><br></div>519‐824‐4120 Ext 56354<br>lelio@uoguelph.ca<br>www.uoguelph.ca/ccs<br>Room 037, Animal Science and Nutrition Building<br>Guelph, Ontario, N1G 2W1<span name="x"></span><br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Andy Carse" <andy.carse@gmail.com><br><b>To: </b>"Lelio Fulgenzi" <lelio@uoguelph.ca><br><b>Cc: </b>"Ryan Huff" <ryanhuff@outlook.com>, "Cisco VoIP List" <cisco-voip@puck.nether.net><br><b>Sent: </b>Wednesday, March 2, 2016 2:27:44 PM<br><b>Subject: </b>Re: [cisco-voip] CUCM Upgrade woes<br><div><br></div><div dir="ltr"><div><div><div>Yes I spotted that after I pasted,<br></div>I'm not sure how it installed v9.1 with that stratum.<br></div>I've changed it now to be a 3 but I guess that its too late.<br></div>I'll try a reboot and see if that fixes it <br></div><div class="gmail_extra"><br><div class="gmail_quote">On 2 March 2016 at 19:23, Lelio Fulgenzi <span dir="ltr"><<a href="mailto:lelio@uoguelph.ca" target="_blank" data-mce-href="mailto:lelio@uoguelph.ca">lelio@uoguelph.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div><div style="font-family:Verdana;font-size:10pt;color:#000000" data-mce-style="font-family: Verdana; font-size: 10pt; color: #000000;"><div><br></div><div>I could be wrong, but it looks like your NTP is not synchronizing properly. It may not be the issue, but it certainly doesn't help.</div><div><br></div><div><br></div><div>Here's a sample of ours with what I think it should look like.</div><div><br></div><div><hr><p style="margin:0px" data-mce-style="margin: 0px;"><span style="font-family:'lucida console',sans-serif" data-mce-style="font-family: 'lucida console',sans-serif;">ntpd (pid 6691) is running...</span></p><p style="margin:0px" data-mce-style="margin: 0px;"><span class=""><span style="font-family:'lucida console',sans-serif" data-mce-style="font-family: 'lucida console',sans-serif;">remote refid st t when poll reach delay offset jitter</span><br><span style="font-family:'lucida console',sans-serif" data-mce-style="font-family: 'lucida console',sans-serif;">==============================================================================</span><br></span><span style="font-family:'lucida console',sans-serif" data-mce-style="font-family: 'lucida console',sans-serif;"> 127.127.1.0 .LOCL. 10 l 2 64 377 0.000 0.000 0.001</span><br><span style="font-family:'lucida console',sans-serif" data-mce-style="font-family: 'lucida console',sans-serif;">*xxx.xxx.xxx.201 xxx.xxx.xxx.53 3 u 721 1024 377 0.663 -0.256 0.124</span><br><span style="font-family:'lucida console',sans-serif" data-mce-style="font-family: 'lucida console',sans-serif;">+xxx.xxx.xxx.201 xxx.xxx.xxx.53 3 u 1021 1024 377 0.924 0.528 0.014</span></p><p style="margin:0px" data-mce-style="margin: 0px;"><br><span style="font-family:'lucida console',sans-serif" data-mce-style="font-family: 'lucida console',sans-serif;">synchronised to NTP server (xxx.xxx.xxx.201) at stratum 4</span><br><span style="font-family:'lucida console',sans-serif" data-mce-style="font-family: 'lucida console',sans-serif;"> time correct to within 86 ms</span><br><span style="font-family:'lucida console',sans-serif" data-mce-style="font-family: 'lucida console',sans-serif;"> polling server every 1024 s</span></p><hr><p style="margin:0px" data-mce-style="margin: 0px;"><br></p><p style="margin:0px" data-mce-style="margin: 0px;"><br></p></div><div><span></span>---<br>Lelio Fulgenzi, B.A.<br>Senior Analyst, Network Infrastructure<br>Computing and Communications Services (CCS)<br>University of Guelph<br><div><br></div>519‐824‐4120 Ext 56354<br><a href="mailto:lelio@uoguelph.ca" target="_blank" data-mce-href="mailto:lelio@uoguelph.ca">lelio@uoguelph.ca</a><br><a href="http://www.uoguelph.ca/ccs" target="_blank" data-mce-href="http://www.uoguelph.ca/ccs">www.uoguelph.ca/ccs</a><br>Room 037, Animal Science and Nutrition Building<br>Guelph, Ontario, N1G 2W1<span></span><br></div><div><br></div><hr><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Andy Carse" <<a href="mailto:andy.carse@gmail.com" target="_blank" data-mce-href="mailto:andy.carse@gmail.com">andy.carse@gmail.com</a>><br><b>To: </b>"Ryan Huff" <<a href="mailto:ryanhuff@outlook.com" target="_blank" data-mce-href="mailto:ryanhuff@outlook.com">ryanhuff@outlook.com</a>><span class=""><br><b>Cc: </b>"Cisco VoIP List" <<a href="mailto:cisco-voip@puck.nether.net" target="_blank" data-mce-href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a>><br></span><b>Sent: </b>Wednesday, March 2, 2016 2:10:31 PM<span class=""><span class=""><br><b>Subject: </b>Re: [cisco-voip] CUCM Upgrade woes<br></span></span><div><br></div><div><div class="h5"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div>.The upgrade was from 9.1.2 to 10.5.2.13900-12.<br></div></div>There where some issues with having the GBNP installed so a direct upgrade was a non-starter.<br></div>This is a hardware refresh and software upgrade rolled up into one project.<br><div><br></div></div>So I backed up the 9.1 <br></div>installed 9.1 on the new hardware to the same specifications, IP addresses and OVA etc<br></div>Restored on to the new publisher ok.<br></div>installed the cop files as required, then had fun with the GBNP.<br></div>So I exported everything except route patterns from the production system.<br></div>Rebuilt another 9.1 this time not installing GBNP.<br></div>Imported into this new cluster.<br></div>Upgraded to 10.5.2.10000-5.<br></div>Then upgraded to 10.5.13900-12.<br></div>Didn't seem to be an issue then came in today and its broken.<br><div><br></div></div>The info you requested is pasted below<br><div><br></div>admin:file view activelog platform/log/diag1.log<br><div><br></div>03-02-2016 18:46:31 Diagnostics Version: 1.0.0<br>03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel]<br>03-02-2016 18:46:32 Hardware Model: VMware<br>03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"]<br>03-02-2016 18:46:32 Version: 10.5.2<br>03-02-2016 18:46:33 disk_space: Is valid module: True<br>03-02-2016 18:46:33 disk_files: Is valid module: True<br>03-02-2016 18:46:33 service_manager: Is valid module: True<br>03-02-2016 18:46:33 tomcat: Is valid module: True<br>03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True<br>03-02-2016 18:46:33 tomcat_keystore: Is valid module: True<br>03-02-2016 18:46:33 tomcat_connectors: Is valid module: True<br>03-02-2016 18:46:33 tomcat_threads: Is valid module: True<br>03-02-2016 18:46:33 tomcat_memory: Is valid module: True<br>03-02-2016 18:46:33 tomcat_sessions: Is valid module: True<br>03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True<br>03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml<br>03-02-2016 18:46:33 validate_network: val: true<br>03-02-2016 18:46:33 validate_network: Is valid module: True<br>03-02-2016 18:46:33 validate_network_adv: Is valid module: False<br><div><br></div>options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : <br><div><br></div><br>admin:<br>admin:file view activelog platform/log/diag1.log<br><div><br></div>03-02-2016 18:46:31 Diagnostics Version: 1.0.0<br>03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel]<br>03-02-2016 18:46:32 Hardware Model: VMware<br>03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"]<br>03-02-2016 18:46:32 Version: 10.5.2<br>03-02-2016 18:46:33 disk_space: Is valid module: True<br>03-02-2016 18:46:33 disk_files: Is valid module: True<br>03-02-2016 18:46:33 service_manager: Is valid module: True<br>03-02-2016 18:46:33 tomcat: Is valid module: True<br>03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True<br>03-02-2016 18:46:33 tomcat_keystore: Is valid module: True<br>03-02-2016 18:46:33 tomcat_connectors: Is valid module: True<br>03-02-2016 18:46:33 tomcat_threads: Is valid module: True<br>03-02-2016 18:46:33 tomcat_memory: Is valid module: True<br>03-02-2016 18:46:33 tomcat_sessions: Is valid module: True<br>03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True<br>03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml<br>03-02-2016 18:46:33 validate_network: val: true<br>03-02-2016 18:46:33 validate_network: Is valid module: True<br>03-02-2016 18:46:33 validate_network_adv: Is valid module: False<br><div><br></div>options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : <br>03-02-2016 18:46:33 raid: getting cpu speed [/usr/local/bin/base_scripts/sd_hwdetect CPUSpeed]<br>03-02-2016 18:46:33 raid: CPU Speed: 2500<br>03-02-2016 18:46:33 raid: model = VMware<br>03-02-2016 18:46:33 raid: Is valid module: True<br>03-02-2016 18:46:33 system_info: Is valid module: True<br>03-02-2016 18:46:33 ntp_reachability: Is valid module: True<br>03-02-2016 18:46:33 ntp_clock_drift: Is valid module: True<br>03-02-2016 18:46:33 ntp_stratum: Is valid module: True<br>03-02-2016 18:46:33 sdl_fragmentation: Is valid module: True<br>03-02-2016 18:46:33 sdi_fragmentation: Is valid module: True<br>03-02-2016 18:46:33 ipv6_networking: IPV6INIT=no<br>03-02-2016 18:46:33 ipv6_networking: IPv6 initialized: no<br>03-02-2016 18:46:33 ipv6_networking: False<br>03-02-2016 18:46:33 ipv6_networking: Is valid module: False<br>03-02-2016 18:46:33 <br>03-02-2016 18:46:33 --> executing test [validate_network], fix: fixauto, stop on error: False<br>03-02-2016 18:46:33 <br>03-02-2016 18:46:33 validate_network: ------------------<br>03-02-2016 18:46:33 validate_network: Testing networking, but skipping duplicate IP test.<br>03-02-2016 18:46:33 validate_network: checking network [/usr/local/bin/base_scripts/validateNetworking.sh -n]<br><div><br></div>options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 54) : <br>03-02-2016 18:46:33 validate_network: retrieving pub name from [/usr/local/platform/conf/platformConfig.xml]<br>03-02-2016 18:46:33 validate_network: Hostname: [XXXXXXXX]<br>03-02-2016 18:46:33 validate_network: found pub name [XXXXXXX]<br>03-02-2016 18:46:33 validate_network: checking /etc/hosts [grep -q `hostname` /etc/hosts]<br>03-02-2016 18:46:33 validate_network: Finding cluster nodes [/usr/local/bin/base_scripts/list_cluster.sh]<br>03-02-2016 18:46:33 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.101>/dev/null]<br>03-02-2016 18:46:35 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.102>/dev/null]<br>03-02-2016 18:46:46 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.4x.101>/dev/null]<br>03-02-2016 18:46:47 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.130>/dev/null]<br>03-02-2016 18:46:48 validate_network: does test script exist [/usr/local/bin/base_scripts/networkDiagnostic.sh]<br>03-02-2016 18:46:48 validate_network: test script exists<br>03-02-2016 18:46:48 validate_network: run network script via expect [./diag_validate_network.exp > /dev/null]<br>03-02-2016 18:46:48 validate_network: result: 0, message: Passed <br><div><br></div><br>end of the file reached<br>options: q=quit, n=next, p=prev, b=begin, e=end (lines 41 - 54 of 54) : <br>admin:<br><div><br></div><br><div><br></div>admin:utils ntp status<br>ntpd (pid 8970) is running...<br><div><br></div> remote refid st t when poll reach delay offset jitter<br>==============================================================================<br> xxx.xxx.55.203 .INIT. 16 u - 1024 0 0.000 0.000 0.000<br>*xxx.xxx.5.203 LOCAL(1) 8 u 268 512 377 0.611 0.304 0.289<br><div><br></div><br>synchronised to NTP server (xxx.xxx.5.203) at stratum 9 <br> time correct to within 37 ms<br> polling server every 512 s<br><div><br></div>Current time in UTC is : Wed Mar 2 18:49:09 UTC 2016<br>Current time in Europe/London is : Wed Mar 2 18:49:09 GMT 2016<br>admin:<br><div><br></div><br><div><br></div>admin:utils ntp server list<br>xxx.xxx.55.203<br><div><br></div>xxx.xxx.5.203<br>admin:<br><div><br></div><br></div>Regards <br></div><div class="gmail_extra"><br><div class="gmail_quote">On 2 March 2016 at 17:36, Ryan Huff <span dir="ltr"><<a href="mailto:ryanhuff@outlook.com" target="_blank" data-mce-href="mailto:ryanhuff@outlook.com">ryanhuff@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;">Not that I'm suggesting you not call TAC but the engineer in me just keeps going ....<br> <br> What (version) did you upgrade from and did you upgrade in-place VMs, DRS/Rebuild or P->V?<br> <br> Do you know if at any point post upgrade, the cluster was healthy and then failed or it has always been in a degraded state since the upgrade?<br> <br> Can you show me the output (from the publisher);<br> <br> - utils diagnose module validate_network<br> - show ntp status<br> - show ntp server list<br> <br> Thanks,<br> <br> Ryan<br><div><div><br> > On Mar 2, 2016, at 12:25 PM, Ryan Huff <<a href="mailto:ryanhuff@outlook.com" target="_blank" data-mce-href="mailto:ryanhuff@outlook.com">ryanhuff@outlook.com</a>> wrote:<br> ><br> > I'd go through a quick checklist while calling in a severity 1 TAC case;<br> ><br> > - forward and reverse DNS for all cluster nodes (and resolving to the correct addresses)<br> ><br> > - verify the processNodes, if using hosts or fqdn, are correctly resolvable. This will prevent A Cisco DB from starting as well as GUI authentication<br> ><br> > - do not have an absurd clock sync on the nodes (Stratum 3 or better)<br> ><br> > Thanks,<br> ><br> > Ryan<br> ><br> >> On Mar 2, 2016, at 12:13 PM, Andy Carse <<a href="mailto:andy.carse@gmail.com" target="_blank" data-mce-href="mailto:andy.carse@gmail.com">andy.carse@gmail.com</a>> wrote:<br> >><br> >> I thought I was home and dry with this upgrade, but it would seem that the gods have deserted me.<br> >><br> >> I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything seemed ok.<br> >> This morning I've come in to find that the database on the publisher won't start.<br> >> So I've tried<br> >> 1. reboot of the cluster (its not gone live yet) no change.<br> >> 2. Utils service start A Cisco DB<br> >> 2. tried dbreplication stop on the subs, then the publisher.<br> >> dbreplication dropddmindb on the subs<br> >> dbreplication dropadmindb on the pub<br> >> The pub comes back with "DropAdminDB cannot be executed on standalone or Cores cluster"<br> >><br> >> I can't even web to ccmadmin on the pub and I forgot to carry out the "Golden Rule" of taking a backup soon after the upgrade.<br> >> If I try to RTM that also fails......<br> >><br> >> Is it time for a start from scratch moment?<br> >><br> >><br> >><br> >> --<br> >> Rgds Andy<br> >><br> >> _______________________________________________<br> >> cisco-voip mailing list<br> >> <a href="mailto:cisco-voip@puck.nether.net" target="_blank" data-mce-href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br> >> <a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank" data-mce-href="https://puck.nether.net/mailman/listinfo/cisco-voip">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br> > _______________________________________________<br> > cisco-voip mailing list<br> > <a href="mailto:cisco-voip@puck.nether.net" target="_blank" data-mce-href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br> > <a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank" data-mce-href="https://puck.nether.net/mailman/listinfo/cisco-voip">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br></div></div></blockquote></div><br><br clear="all"><br>-- <br><div>Rgds Andy<br><div><br></div></div></div><br>_______________________________________________<br>cisco-voip mailing list<br><a href="mailto:cisco-voip@puck.nether.net" target="_blank" data-mce-href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br><a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank" data-mce-href="https://puck.nether.net/mailman/listinfo/cisco-voip">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br></div></div></div><div><br></div></div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">Rgds Andy<br><div><br></div></div></div></div><div><br></div></div></body></html>