[cisco-voip] CUCM Upgrade woes
Andy Carse
andy.carse at gmail.com
Wed Mar 2 14:27:44 EST 2016
Yes I spotted that after I pasted,
I'm not sure how it installed v9.1 with that stratum.
I've changed it now to be a 3 but I guess that its too late.
I'll try a reboot and see if that fixes it
On 2 March 2016 at 19:23, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:
>
> I could be wrong, but it looks like your NTP is not synchronizing
> properly. It may not be the issue, but it certainly doesn't help.
>
>
> Here's a sample of ours with what I think it should look like.
>
> ------------------------------
>
> ntpd (pid 6691) is running...
>
> remote refid st t when poll reach delay offset jitter
>
> ==============================================================================
> 127.127.1.0 .LOCL. 10 l 2 64 377 0.000 0.000 0.001
> *xxx.xxx.xxx.201 xxx.xxx.xxx.53 3 u 721 1024 377 0.663 -0.256 0.124
> +xxx.xxx.xxx.201 xxx.xxx.xxx.53 3 u 1021 1024 377 0.924 0.528 0.014
>
>
> synchronised to NTP server (xxx.xxx.xxx.201) at stratum 4
> time correct to within 86 ms
> polling server every 1024 s
> ------------------------------
>
>
>
> ---
> Lelio Fulgenzi, B.A.
> Senior Analyst, Network Infrastructure
> Computing and Communications Services (CCS)
> University of Guelph
>
> 519‐824‐4120 Ext 56354
> lelio at uoguelph.ca
> www.uoguelph.ca/ccs
> Room 037, Animal Science and Nutrition Building
> Guelph, Ontario, N1G 2W1
>
> ------------------------------
> *From: *"Andy Carse" <andy.carse at gmail.com>
> *To: *"Ryan Huff" <ryanhuff at outlook.com>
> *Cc: *"Cisco VoIP List" <cisco-voip at puck.nether.net>
> *Sent: *Wednesday, March 2, 2016 2:10:31 PM
> *Subject: *Re: [cisco-voip] CUCM Upgrade woes
>
> .The upgrade was from 9.1.2 to 10.5.2.13900-12.
> There where some issues with having the GBNP installed so a direct upgrade
> was a non-starter.
> This is a hardware refresh and software upgrade rolled up into one project.
>
> So I backed up the 9.1
> installed 9.1 on the new hardware to the same specifications, IP addresses
> and OVA etc
> Restored on to the new publisher ok.
> installed the cop files as required, then had fun with the GBNP.
> So I exported everything except route patterns from the production system.
> Rebuilt another 9.1 this time not installing GBNP.
> Imported into this new cluster.
> Upgraded to 10.5.2.10000-5.
> Then upgraded to 10.5.13900-12.
> Didn't seem to be an issue then came in today and its broken.
>
> The info you requested is pasted below
>
> admin:file view activelog platform/log/diag1.log
>
> 03-02-2016 18:46:31 Diagnostics Version: 1.0.0
> 03-02-2016 18:46:31 getting hardware model
> [/usr/local/bin/base_scripts/sd_hwdetect HWModel]
> 03-02-2016 18:46:32 Hardware Model: VMware
> 03-02-2016 18:46:32 getting verson number [rpm -q
> --nodigest --nosignature master | sed -e "s/master-//"]
> 03-02-2016 18:46:32 Version: 10.5.2
> 03-02-2016 18:46:33 disk_space: Is valid module: True
> 03-02-2016 18:46:33 disk_files: Is valid module: True
> 03-02-2016 18:46:33 service_manager: Is valid module: True
> 03-02-2016 18:46:33 tomcat: Is valid module: True
> 03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True
> 03-02-2016 18:46:33 tomcat_keystore: Is valid module: True
> 03-02-2016 18:46:33 tomcat_connectors: Is valid module: True
> 03-02-2016 18:46:33 tomcat_threads: Is valid module: True
> 03-02-2016 18:46:33 tomcat_memory: Is valid module: True
> 03-02-2016 18:46:33 tomcat_sessions: Is valid module: True
> 03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True
> 03-02-2016 18:46:33 validate_network: Product specific XML file:
> /usr/local/platform/conf/cli/cliProduct.xml
> 03-02-2016 18:46:33 validate_network: val: true
> 03-02-2016 18:46:33 validate_network: Is valid module: True
> 03-02-2016 18:46:33 validate_network_adv: Is valid module: False
>
> options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) :
>
>
> admin:
> admin:file view activelog platform/log/diag1.log
>
> 03-02-2016 18:46:31 Diagnostics Version: 1.0.0
> 03-02-2016 18:46:31 getting hardware model
> [/usr/local/bin/base_scripts/sd_hwdetect HWModel]
> 03-02-2016 18:46:32 Hardware Model: VMware
> 03-02-2016 18:46:32 getting verson number [rpm -q
> --nodigest --nosignature master | sed -e "s/master-//"]
> 03-02-2016 18:46:32 Version: 10.5.2
> 03-02-2016 18:46:33 disk_space: Is valid module: True
> 03-02-2016 18:46:33 disk_files: Is valid module: True
> 03-02-2016 18:46:33 service_manager: Is valid module: True
> 03-02-2016 18:46:33 tomcat: Is valid module: True
> 03-02-2016 18:46:33 tomcat_deadlocks: Is valid module: True
> 03-02-2016 18:46:33 tomcat_keystore: Is valid module: True
> 03-02-2016 18:46:33 tomcat_connectors: Is valid module: True
> 03-02-2016 18:46:33 tomcat_threads: Is valid module: True
> 03-02-2016 18:46:33 tomcat_memory: Is valid module: True
> 03-02-2016 18:46:33 tomcat_sessions: Is valid module: True
> 03-02-2016 18:46:33 tomcat_heapdump: Is valid module: True
> 03-02-2016 18:46:33 validate_network: Product specific XML file:
> /usr/local/platform/conf/cli/cliProduct.xml
> 03-02-2016 18:46:33 validate_network: val: true
> 03-02-2016 18:46:33 validate_network: Is valid module: True
> 03-02-2016 18:46:33 validate_network_adv: Is valid module: False
>
> options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) :
> 03-02-2016 18:46:33 raid: getting cpu speed
> [/usr/local/bin/base_scripts/sd_hwdetect CPUSpeed]
> 03-02-2016 18:46:33 raid: CPU Speed: 2500
> 03-02-2016 18:46:33 raid: model = VMware
> 03-02-2016 18:46:33 raid: Is valid module: True
> 03-02-2016 18:46:33 system_info: Is valid module: True
> 03-02-2016 18:46:33 ntp_reachability: Is valid module: True
> 03-02-2016 18:46:33 ntp_clock_drift: Is valid module: True
> 03-02-2016 18:46:33 ntp_stratum: Is valid module: True
> 03-02-2016 18:46:33 sdl_fragmentation: Is valid module: True
> 03-02-2016 18:46:33 sdi_fragmentation: Is valid module: True
> 03-02-2016 18:46:33 ipv6_networking: IPV6INIT=no
> 03-02-2016 18:46:33 ipv6_networking: IPv6 initialized: no
> 03-02-2016 18:46:33 ipv6_networking: False
> 03-02-2016 18:46:33 ipv6_networking: Is valid module: False
> 03-02-2016 18:46:33
> 03-02-2016 18:46:33 --> executing test
> [validate_network], fix: fixauto, stop on error: False
> 03-02-2016 18:46:33
> 03-02-2016 18:46:33 validate_network: ------------------
> 03-02-2016 18:46:33 validate_network: Testing networking, but skipping
> duplicate IP test.
> 03-02-2016 18:46:33 validate_network: checking network
> [/usr/local/bin/base_scripts/validateNetworking.sh -n]
>
> options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 54) :
> 03-02-2016 18:46:33 validate_network: retrieving pub name from
> [/usr/local/platform/conf/platformConfig.xml]
> 03-02-2016 18:46:33 validate_network: Hostname: [XXXXXXXX]
> 03-02-2016 18:46:33 validate_network: found pub name [XXXXXXX]
> 03-02-2016 18:46:33 validate_network: checking /etc/hosts [grep -q
> `hostname` /etc/hosts]
> 03-02-2016 18:46:33 validate_network: Finding cluster nodes
> [/usr/local/bin/base_scripts/list_cluster.sh]
> 03-02-2016 18:46:33 validate_network: running
> [./diag_validate_network_sftp.exp sftpuser at xxx.xxx.9x.101>/dev/null]
> 03-02-2016 18:46:35 validate_network: running
> [./diag_validate_network_sftp.exp sftpuser at xxx.xxx.9x.102>/dev/null]
> 03-02-2016 18:46:46 validate_network: running
> [./diag_validate_network_sftp.exp sftpuser at xxx.xxx.4x.101>/dev/null]
> 03-02-2016 18:46:47 validate_network: running
> [./diag_validate_network_sftp.exp sftpuser at xxx.xxx.9x.130>/dev/null]
> 03-02-2016 18:46:48 validate_network: does test script exist
> [/usr/local/bin/base_scripts/networkDiagnostic.sh]
> 03-02-2016 18:46:48 validate_network: test script exists
> 03-02-2016 18:46:48 validate_network: run network script via expect
> [./diag_validate_network.exp > /dev/null]
> 03-02-2016 18:46:48 validate_network: result: 0, message: Passed
>
>
> end of the file reached
> options: q=quit, n=next, p=prev, b=begin, e=end (lines 41 - 54 of 54) :
> admin:
>
>
>
> admin:utils ntp status
> ntpd (pid 8970) is running...
>
> remote refid st t when poll reach delay offset
> jitter
>
> ==============================================================================
> xxx.xxx.55.203 .INIT. 16 u - 1024 0 0.000 0.000
> 0.000
> *xxx.xxx.5.203 LOCAL(1) 8 u 268 512 377 0.611 0.304
> 0.289
>
>
> synchronised to NTP server (xxx.xxx.5.203) at stratum 9
> time correct to within 37 ms
> polling server every 512 s
>
> Current time in UTC is : Wed Mar 2 18:49:09 UTC 2016
> Current time in Europe/London is : Wed Mar 2 18:49:09 GMT 2016
> admin:
>
>
>
> admin:utils ntp server list
> xxx.xxx.55.203
>
> xxx.xxx.5.203
> admin:
>
>
> Regards
>
> On 2 March 2016 at 17:36, Ryan Huff <ryanhuff at outlook.com> wrote:
>
>> Not that I'm suggesting you not call TAC but the engineer in me just
>> keeps going ....
>>
>> What (version) did you upgrade from and did you upgrade in-place VMs,
>> DRS/Rebuild or P->V?
>>
>> Do you know if at any point post upgrade, the cluster was healthy and
>> then failed or it has always been in a degraded state since the upgrade?
>>
>> Can you show me the output (from the publisher);
>>
>> - utils diagnose module validate_network
>> - show ntp status
>> - show ntp server list
>>
>> Thanks,
>>
>> Ryan
>>
>> > On Mar 2, 2016, at 12:25 PM, Ryan Huff <ryanhuff at outlook.com> wrote:
>> >
>> > I'd go through a quick checklist while calling in a severity 1 TAC case;
>> >
>> > - forward and reverse DNS for all cluster nodes (and resolving to the
>> correct addresses)
>> >
>> > - verify the processNodes, if using hosts or fqdn, are correctly
>> resolvable. This will prevent A Cisco DB from starting as well as GUI
>> authentication
>> >
>> > - do not have an absurd clock sync on the nodes (Stratum 3 or better)
>> >
>> > Thanks,
>> >
>> > Ryan
>> >
>> >> On Mar 2, 2016, at 12:13 PM, Andy Carse <andy.carse at gmail.com> wrote:
>> >>
>> >> I thought I was home and dry with this upgrade, but it would seem that
>> the gods have deserted me.
>> >>
>> >> I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything
>> seemed ok.
>> >> This morning I've come in to find that the database on the publisher
>> won't start.
>> >> So I've tried
>> >> 1. reboot of the cluster (its not gone live yet) no change.
>> >> 2. Utils service start A Cisco DB
>> >> 2. tried dbreplication stop on the subs, then the publisher.
>> >> dbreplication dropddmindb on the subs
>> >> dbreplication dropadmindb on the pub
>> >> The pub comes back with "DropAdminDB cannot be executed on standalone
>> or Cores cluster"
>> >>
>> >> I can't even web to ccmadmin on the pub and I forgot to carry out the
>> "Golden Rule" of taking a backup soon after the upgrade.
>> >> If I try to RTM that also fails......
>> >>
>> >> Is it time for a start from scratch moment?
>> >>
>> >>
>> >>
>> >> --
>> >> Rgds Andy
>> >>
>> >> _______________________________________________
>> >> cisco-voip mailing list
>> >> cisco-voip at puck.nether.net
>> >> https://puck.nether.net/mailman/listinfo/cisco-voip
>> > _______________________________________________
>> > cisco-voip mailing list
>> > cisco-voip at puck.nether.net
>> > https://puck.nether.net/mailman/listinfo/cisco-voip
>>
>
>
>
> --
> Rgds Andy
>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
--
Rgds Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20160302/16b1639f/attachment.html>
More information about the cisco-voip
mailing list