[cisco-voip] CUCM Upgrade woes

Lelio Fulgenzi lelio at uoguelph.ca
Wed Mar 2 14:33:49 EST 2016


i'd first try getting NTP to work first on all the hosts. 

a reboot might fix things, but it might not. sometimes, the database sync needs to be restarted manually and won't restart even after a restart of the cluster. 

i'd engage the TAC after the NTP is fixed. 


--- 
Lelio Fulgenzi, B.A. 
Senior Analyst, Network Infrastructure 
Computing and Communications Services (CCS) 
University of Guelph 

519‐824‐4120 Ext 56354 
lelio at uoguelph.ca 
www.uoguelph.ca/ccs 
Room 037, Animal Science and Nutrition Building 
Guelph, Ontario, N1G 2W1 

----- Original Message -----

From: "Andy Carse" <andy.carse at gmail.com> 
To: "Lelio Fulgenzi" <lelio at uoguelph.ca> 
Cc: "Ryan Huff" <ryanhuff at outlook.com>, "Cisco VoIP List" <cisco-voip at puck.nether.net> 
Sent: Wednesday, March 2, 2016 2:27:44 PM 
Subject: Re: [cisco-voip] CUCM Upgrade woes 

Yes I spotted that after I pasted, 
I'm not sure how it installed v9.1 with that stratum. 
I've changed it now to be a 3 but I guess that its too late. 
I'll try a reboot and see if that fixes it 

On 2 March 2016 at 19:23, Lelio Fulgenzi < lelio at uoguelph.ca > wrote: 




I could be wrong, but it looks like your NTP is not synchronizing properly. It may not be the issue, but it certainly doesn't help. 


Here's a sample of ours with what I think it should look like. 




ntpd (pid 6691) is running... 

remote refid st t when poll reach delay offset jitter 
============================================================================== 
127.127.1.0 .LOCL. 10 l 2 64 377 0.000 0.000 0.001 
*xxx.xxx.xxx.201 xxx.xxx.xxx.53 3 u 721 1024 377 0.663 -0.256 0.124 
+xxx.xxx.xxx.201 xxx.xxx.xxx.53 3 u 1021 1024 377 0.924 0.528 0.014 


synchronised to NTP server (xxx.xxx.xxx.201) at stratum 4 
time correct to within 86 ms 
polling server every 1024 s 






--- 
Lelio Fulgenzi, B.A. 
Senior Analyst, Network Infrastructure 
Computing and Communications Services (CCS) 
University of Guelph 

519‐824‐4120 Ext 56354 
lelio at uoguelph.ca 
www.uoguelph.ca/ccs 
Room 037, Animal Science and Nutrition Building 
Guelph, Ontario, N1G 2W1 


From: "Andy Carse" < andy.carse at gmail.com > 
To: "Ryan Huff" < ryanhuff at outlook.com > 
Cc: "Cisco VoIP List" < cisco-voip at puck.nether.net > 
Sent: Wednesday, March 2, 2016 2:10:31 PM 
Subject: Re: [cisco-voip] CUCM Upgrade woes 

.The upgrade was from 9.1.2 to 10.5.2.13900-12. 
There where some issues with having the GBNP installed so a direct upgrade was a non-starter. 
This is a hardware refresh and software upgrade rolled up into one project. 

So I backed up the 9.1 
installed 9.1 on the new hardware to the same specifications, IP addresses and OVA etc 
Restored on to the new publisher ok. 
installed the cop files as required, then had fun with the GBNP. 
So I exported everything except route patterns from the production system. 
Rebuilt another 9.1 this time not installing GBNP. 
Imported into this new cluster. 
Upgraded to 10.5.2.10000-5. 
Then upgraded to 10.5.13900-12. 
Didn't seem to be an issue then came in today and its broken. 

The info you requested is pasted below 

admin:file view activelog platform/log/diag1.log 

03-02-2016 18:46:31 Diagnostics Version: 1.0.0 
03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel] 
03-02-2016 18:46:32 Hardware Model: VMware 
03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"] 
03-02-2016 18:46:32 Version: 10.5.2 
03-02-2016 18:46:33 disk_space: Is valid module: True 
03-02-2016 18:46:33 disk_files: Is valid module: True 
03-02-2016 18:46:33 service_manager: Is valid module: True 
03-02-2016 18:46:33 tomcat: Is valid module: True 
03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True 
03-02-2016 18:46:33 tomcat_keystore: Is valid module: True 
03-02-2016 18:46:33 tomcat_connectors: Is valid module: True 
03-02-2016 18:46:33 tomcat_threads: Is valid module: True 
03-02-2016 18:46:33 tomcat_memory: Is valid module: True 
03-02-2016 18:46:33 tomcat_sessions: Is valid module: True 
03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True 
03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml 
03-02-2016 18:46:33 validate_network: val: true 
03-02-2016 18:46:33 validate_network: Is valid module: True 
03-02-2016 18:46:33 validate_network_adv: Is valid module: False 

options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : 


admin: 
admin:file view activelog platform/log/diag1.log 

03-02-2016 18:46:31 Diagnostics Version: 1.0.0 
03-02-2016 18:46:31 getting hardware model [/usr/local/bin/base_scripts/sd_hwdetect HWModel] 
03-02-2016 18:46:32 Hardware Model: VMware 
03-02-2016 18:46:32 getting verson number [rpm -q --nodigest --nosignature master | sed -e "s/master-//"] 
03-02-2016 18:46:32 Version: 10.5.2 
03-02-2016 18:46:33 disk_space: Is valid module: True 
03-02-2016 18:46:33 disk_files: Is valid module: True 
03-02-2016 18:46:33 service_manager: Is valid module: True 
03-02-2016 18:46:33 tomcat: Is valid module: True 
03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True 
03-02-2016 18:46:33 tomcat_keystore: Is valid module: True 
03-02-2016 18:46:33 tomcat_connectors: Is valid module: True 
03-02-2016 18:46:33 tomcat_threads: Is valid module: True 
03-02-2016 18:46:33 tomcat_memory: Is valid module: True 
03-02-2016 18:46:33 tomcat_sessions: Is valid module: True 
03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True 
03-02-2016 18:46:33 validate_network: Product specific XML file: /usr/local/platform/conf/cli/cliProduct.xml 
03-02-2016 18:46:33 validate_network: val: true 
03-02-2016 18:46:33 validate_network: Is valid module: True 
03-02-2016 18:46:33 validate_network_adv: Is valid module: False 

options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) : 
03-02-2016 18:46:33 raid: getting cpu speed [/usr/local/bin/base_scripts/sd_hwdetect CPUSpeed] 
03-02-2016 18:46:33 raid: CPU Speed: 2500 
03-02-2016 18:46:33 raid: model = VMware 
03-02-2016 18:46:33 raid: Is valid module: True 
03-02-2016 18:46:33 system_info: Is valid module: True 
03-02-2016 18:46:33 ntp_reachability: Is valid module: True 
03-02-2016 18:46:33 ntp_clock_drift: Is valid module: True 
03-02-2016 18:46:33 ntp_stratum: Is valid module: True 
03-02-2016 18:46:33 sdl_fragmentation: Is valid module: True 
03-02-2016 18:46:33 sdi_fragmentation: Is valid module: True 
03-02-2016 18:46:33 ipv6_networking: IPV6INIT=no 
03-02-2016 18:46:33 ipv6_networking: IPv6 initialized: no 
03-02-2016 18:46:33 ipv6_networking: False 
03-02-2016 18:46:33 ipv6_networking: Is valid module: False 
03-02-2016 18:46:33 
03-02-2016 18:46:33 --> executing test [validate_network], fix: fixauto, stop on error: False 
03-02-2016 18:46:33 
03-02-2016 18:46:33 validate_network: ------------------ 
03-02-2016 18:46:33 validate_network: Testing networking, but skipping duplicate IP test. 
03-02-2016 18:46:33 validate_network: checking network [/usr/local/bin/base_scripts/validateNetworking.sh -n] 

options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 54) : 
03-02-2016 18:46:33 validate_network: retrieving pub name from [/usr/local/platform/conf/platformConfig.xml] 
03-02-2016 18:46:33 validate_network: Hostname: [XXXXXXXX] 
03-02-2016 18:46:33 validate_network: found pub name [XXXXXXX] 
03-02-2016 18:46:33 validate_network: checking /etc/hosts [grep -q `hostname` /etc/hosts] 
03-02-2016 18:46:33 validate_network: Finding cluster nodes [/usr/local/bin/base_scripts/list_cluster.sh] 
03-02-2016 18:46:33 validate_network: running [./diag_validate_network_sftp.exp sftpuser at xxx.xxx.9x.101>/dev/null] 
03-02-2016 18:46:35 validate_network: running [./diag_validate_network_sftp.exp sftpuser at xxx.xxx.9x.102>/dev/null] 
03-02-2016 18:46:46 validate_network: running [./diag_validate_network_sftp.exp sftpuser at xxx.xxx.4x.101>/dev/null] 
03-02-2016 18:46:47 validate_network: running [./diag_validate_network_sftp.exp sftpuser at xxx.xxx.9x.130>/dev/null] 
03-02-2016 18:46:48 validate_network: does test script exist [/usr/local/bin/base_scripts/networkDiagnostic.sh] 
03-02-2016 18:46:48 validate_network: test script exists 
03-02-2016 18:46:48 validate_network: run network script via expect [./diag_validate_network.exp > /dev/null] 
03-02-2016 18:46:48 validate_network: result: 0, message: Passed 


end of the file reached 
options: q=quit, n=next, p=prev, b=begin, e=end (lines 41 - 54 of 54) : 
admin: 



admin:utils ntp status 
ntpd (pid 8970) is running... 

remote refid st t when poll reach delay offset jitter 
============================================================================== 
xxx.xxx.55.203 .INIT. 16 u - 1024 0 0.000 0.000 0.000 
*xxx.xxx.5.203 LOCAL(1) 8 u 268 512 377 0.611 0.304 0.289 


synchronised to NTP server (xxx.xxx.5.203) at stratum 9 
time correct to within 37 ms 
polling server every 512 s 

Current time in UTC is : Wed Mar 2 18:49:09 UTC 2016 
Current time in Europe/London is : Wed Mar 2 18:49:09 GMT 2016 
admin: 



admin:utils ntp server list 
xxx.xxx.55.203 

xxx.xxx.5.203 
admin: 


Regards 

On 2 March 2016 at 17:36, Ryan Huff < ryanhuff at outlook.com > wrote: 

<blockquote>
Not that I'm suggesting you not call TAC but the engineer in me just keeps going .... 

What (version) did you upgrade from and did you upgrade in-place VMs, DRS/Rebuild or P->V? 

Do you know if at any point post upgrade, the cluster was healthy and then failed or it has always been in a degraded state since the upgrade? 

Can you show me the output (from the publisher); 

- utils diagnose module validate_network 
- show ntp status 
- show ntp server list 

Thanks, 

Ryan 

> On Mar 2, 2016, at 12:25 PM, Ryan Huff < ryanhuff at outlook.com > wrote: 
> 
> I'd go through a quick checklist while calling in a severity 1 TAC case; 
> 
> - forward and reverse DNS for all cluster nodes (and resolving to the correct addresses) 
> 
> - verify the processNodes, if using hosts or fqdn, are correctly resolvable. This will prevent A Cisco DB from starting as well as GUI authentication 
> 
> - do not have an absurd clock sync on the nodes (Stratum 3 or better) 
> 
> Thanks, 
> 
> Ryan 
> 
>> On Mar 2, 2016, at 12:13 PM, Andy Carse < andy.carse at gmail.com > wrote: 
>> 
>> I thought I was home and dry with this upgrade, but it would seem that the gods have deserted me. 
>> 
>> I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything seemed ok. 
>> This morning I've come in to find that the database on the publisher won't start. 
>> So I've tried 
>> 1. reboot of the cluster (its not gone live yet) no change. 
>> 2. Utils service start A Cisco DB 
>> 2. tried dbreplication stop on the subs, then the publisher. 
>> dbreplication dropddmindb on the subs 
>> dbreplication dropadmindb on the pub 
>> The pub comes back with "DropAdminDB cannot be executed on standalone or Cores cluster" 
>> 
>> I can't even web to ccmadmin on the pub and I forgot to carry out the "Golden Rule" of taking a backup soon after the upgrade. 
>> If I try to RTM that also fails...... 
>> 
>> Is it time for a start from scratch moment? 
>> 
>> 
>> 
>> -- 
>> Rgds Andy 
>> 
>> _______________________________________________ 
>> cisco-voip mailing list 
>> cisco-voip at puck.nether.net 
>> https://puck.nether.net/mailman/listinfo/cisco-voip 
> _______________________________________________ 
> cisco-voip mailing list 
> cisco-voip at puck.nether.net 
> https://puck.nether.net/mailman/listinfo/cisco-voip 






-- 
Rgds Andy 


_______________________________________________ 
cisco-voip mailing list 
cisco-voip at puck.nether.net 
https://puck.nether.net/mailman/listinfo/cisco-voip 


</blockquote>




-- 
Rgds Andy 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20160302/50609c10/attachment.html>


More information about the cisco-voip mailing list