<div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div><div><div><div>.The upgrade was from 9.1.2 to 10.5.2.13900-12.<br></div></div>There where some issues with having the GBNP installed so a direct upgrade was a non-starter.<br></div>This is a hardware refresh and software upgrade rolled up into one project.<br><br></div>So I backed up the 9.1 <br></div>installed 9.1 on the new hardware to the same specifications, IP addresses and OVA etc<br></div>Restored on to the new publisher ok.<br></div>installed the cop files as required, then had fun with the GBNP.<br></div>So I exported everything except route patterns from the production system.<br></div>Rebuilt another 9.1 this time not installing GBNP.<br></div>Imported into this new cluster.<br></div>Upgraded to 10.5.2.10000-5.<br></div>Then upgraded to 10.5.13900-12.<br></div>Didn't seem to be an issue then came in today and its broken.<br><br></div>The info you requested is pasted below<br><br>admin:file view activelog platform/log/diag1.log<br><br>03-02-2016 18:46:31 Diagnostics Version: 1.0.0<br>03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel]<br>03-02-2016 18:46:32 Hardware Model: VMware<br>03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"]<br>03-02-2016 18:46:32 Version: 10.5.2<br>03-02-2016 18:46:33 disk_space: Is valid module: True<br>03-02-2016 18:46:33 disk_files: Is valid module: True<br>03-02-2016 18:46:33 service_manager: Is valid module: True<br>03-02-2016 18:46:33 tomcat: Is valid module: True<br>03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True<br>03-02-2016 18:46:33 tomcat_keystore: Is valid module: True<br>03-02-2016 18:46:33 tomcat_connectors: Is valid module: True<br>03-02-2016 18:46:33 tomcat_threads: Is valid module: True<br>03-02-2016 18:46:33 tomcat_memory: Is valid module: True<br>03-02-2016 18:46:33 tomcat_sessions: Is valid module: True<br>03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True<br>03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml<br>03-02-2016 18:46:33 validate_network: val: true<br>03-02-2016 18:46:33 validate_network: Is valid module: True<br>03-02-2016 18:46:33 validate_network_adv: Is valid module: False<br><br>options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : <br><br><br>admin:<br>admin:file view activelog platform/log/diag1.log<br><br>03-02-2016 18:46:31 Diagnostics Version: 1.0.0<br>03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel]<br>03-02-2016 18:46:32 Hardware Model: VMware<br>03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"]<br>03-02-2016 18:46:32 Version: 10.5.2<br>03-02-2016 18:46:33 disk_space: Is valid module: True<br>03-02-2016 18:46:33 disk_files: Is valid module: True<br>03-02-2016 18:46:33 service_manager: Is valid module: True<br>03-02-2016 18:46:33 tomcat: Is valid module: True<br>03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True<br>03-02-2016 18:46:33 tomcat_keystore: Is valid module: True<br>03-02-2016 18:46:33 tomcat_connectors: Is valid module: True<br>03-02-2016 18:46:33 tomcat_threads: Is valid module: True<br>03-02-2016 18:46:33 tomcat_memory: Is valid module: True<br>03-02-2016 18:46:33 tomcat_sessions: Is valid module: True<br>03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True<br>03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml<br>03-02-2016 18:46:33 validate_network: val: true<br>03-02-2016 18:46:33 validate_network: Is valid module: True<br>03-02-2016 18:46:33 validate_network_adv: Is valid module: False<br><br>options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : <br>03-02-2016 18:46:33 raid: getting cpu speed [/usr/local/bin/base_scripts/sd_hwdetect CPUSpeed]<br>03-02-2016 18:46:33 raid: CPU Speed: 2500<br>03-02-2016 18:46:33 raid: model = VMware<br>03-02-2016 18:46:33 raid: Is valid module: True<br>03-02-2016 18:46:33 system_info: Is valid module: True<br>03-02-2016 18:46:33 ntp_reachability: Is valid module: True<br>03-02-2016 18:46:33 ntp_clock_drift: Is valid module: True<br>03-02-2016 18:46:33 ntp_stratum: Is valid module: True<br>03-02-2016 18:46:33 sdl_fragmentation: Is valid module: True<br>03-02-2016 18:46:33 sdi_fragmentation: Is valid module: True<br>03-02-2016 18:46:33 ipv6_networking: IPV6INIT=no<br>03-02-2016 18:46:33 ipv6_networking: IPv6 initialized: no<br>03-02-2016 18:46:33 ipv6_networking: False<br>03-02-2016 18:46:33 ipv6_networking: Is valid module: False<br>03-02-2016 18:46:33 <br>03-02-2016 18:46:33 --> executing test [validate_network], fix: fixauto, stop on error: False<br>03-02-2016 18:46:33 <br>03-02-2016 18:46:33 validate_network: ------------------<br>03-02-2016 18:46:33 validate_network: Testing networking, but skipping duplicate IP test.<br>03-02-2016 18:46:33 validate_network: checking network [/usr/local/bin/base_scripts/validateNetworking.sh -n]<br><br>options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 54) : <br>03-02-2016 18:46:33 validate_network: retrieving pub name from [/usr/local/platform/conf/platformConfig.xml]<br>03-02-2016 18:46:33 validate_network: Hostname: [XXXXXXXX]<br>03-02-2016 18:46:33 validate_network: found pub name [XXXXXXX]<br>03-02-2016 18:46:33 validate_network: checking /etc/hosts [grep -q `hostname` /etc/hosts]<br>03-02-2016 18:46:33 validate_network: Finding cluster nodes [/usr/local/bin/base_scripts/list_cluster.sh]<br>03-02-2016 18:46:33 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.101>/dev/null]<br>03-02-2016 18:46:35 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.102>/dev/null]<br>03-02-2016 18:46:46 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.4x.101>/dev/null]<br>03-02-2016 18:46:47 validate_network: running [./diag_validate_network_sftp.exp sftpuser@xxx.xxx.9x.130>/dev/null]<br>03-02-2016 18:46:48 validate_network: does test script exist [/usr/local/bin/base_scripts/networkDiagnostic.sh]<br>03-02-2016 18:46:48 validate_network: test script exists<br>03-02-2016 18:46:48 validate_network: run network script via expect [./diag_validate_network.exp > /dev/null]<br>03-02-2016 18:46:48 validate_network: result: 0, message: Passed <br><br><br>end of the file reached<br>options: q=quit, n=next, p=prev, b=begin, e=end (lines 41 - 54 of 54) : <br>admin:<br><br><br><br>admin:utils ntp status<br>ntpd (pid 8970) is running...<br><br> remote refid st t when poll reach delay offset jitter<br>==============================================================================<br> xxx.xxx.55.203 .INIT. 16 u - 1024 0 0.000 0.000 0.000<br>*xxx.xxx.5.203 LOCAL(1) 8 u 268 512 377 0.611 0.304 0.289<br><br><br>synchronised to NTP server (xxx.xxx.5.203) at stratum 9 <br> time correct to within 37 ms<br> polling server every 512 s<br><br>Current time in UTC is : Wed Mar 2 18:49:09 UTC 2016<br>Current time in Europe/London is : Wed Mar 2 18:49:09 GMT 2016<br>admin:<br><br><br><br>admin:utils ntp server list<br>xxx.xxx.55.203<br><br>xxx.xxx.5.203<br>admin:<br><br><br></div>Regards <br></div><div class="gmail_extra"><br><div class="gmail_quote">On 2 March 2016 at 17:36, Ryan Huff <span dir="ltr"><<a href="mailto:ryanhuff@outlook.com" target="_blank">ryanhuff@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Not that I'm suggesting you not call TAC but the engineer in me just keeps going ....<br>
<br>
What (version) did you upgrade from and did you upgrade in-place VMs, DRS/Rebuild or P->V?<br>
<br>
Do you know if at any point post upgrade, the cluster was healthy and then failed or it has always been in a degraded state since the upgrade?<br>
<br>
Can you show me the output (from the publisher);<br>
<br>
- utils diagnose module validate_network<br>
- show ntp status<br>
- show ntp server list<br>
<br>
Thanks,<br>
<br>
Ryan<br>
<div class="HOEnZb"><div class="h5"><br>
> On Mar 2, 2016, at 12:25 PM, Ryan Huff <<a href="mailto:ryanhuff@outlook.com">ryanhuff@outlook.com</a>> wrote:<br>
><br>
> I'd go through a quick checklist while calling in a severity 1 TAC case;<br>
><br>
> - forward and reverse DNS for all cluster nodes (and resolving to the correct addresses)<br>
><br>
> - verify the processNodes, if using hosts or fqdn, are correctly resolvable. This will prevent A Cisco DB from starting as well as GUI authentication<br>
><br>
> - do not have an absurd clock sync on the nodes (Stratum 3 or better)<br>
><br>
> Thanks,<br>
><br>
> Ryan<br>
><br>
>> On Mar 2, 2016, at 12:13 PM, Andy Carse <<a href="mailto:andy.carse@gmail.com">andy.carse@gmail.com</a>> wrote:<br>
>><br>
>> I thought I was home and dry with this upgrade, but it would seem that the gods have deserted me.<br>
>><br>
>> I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything seemed ok.<br>
>> This morning I've come in to find that the database on the publisher won't start.<br>
>> So I've tried<br>
>> 1. reboot of the cluster (its not gone live yet) no change.<br>
>> 2. Utils service start A Cisco DB<br>
>> 2. tried dbreplication stop on the subs, then the publisher.<br>
>> dbreplication dropddmindb on the subs<br>
>> dbreplication dropadmindb on the pub<br>
>> The pub comes back with "DropAdminDB cannot be executed on standalone or Cores cluster"<br>
>><br>
>> I can't even web to ccmadmin on the pub and I forgot to carry out the "Golden Rule" of taking a backup soon after the upgrade.<br>
>> If I try to RTM that also fails......<br>
>><br>
>> Is it time for a start from scratch moment?<br>
>><br>
>><br>
>><br>
>> --<br>
>> Rgds Andy<br>
>><br>
>> _______________________________________________<br>
>> cisco-voip mailing list<br>
>> <a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
>> <a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
> _______________________________________________<br>
> cisco-voip mailing list<br>
> <a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">Rgds Andy<br><br></div>
</div>