[cisco-voip] CUCM Upgrade woes

Andy Carse andy.carse at gmail.com
Wed Mar 2 14:10:31 EST 2016 

.The upgrade was from 9.1.2 to 10.5.2.13900-12.
There where some issues with having the GBNP installed so a direct upgrade
was a non-starter.
This is a hardware refresh and software upgrade rolled up into one project.

So I backed up the 9.1
installed 9.1 on the new hardware to the same specifications, IP addresses
and OVA etc
Restored on to the new publisher ok.
installed the cop files as required, then had fun with the GBNP.
So I exported everything except route patterns from the production system.
Rebuilt another 9.1 this time not installing GBNP.
Imported into this new cluster.
Upgraded to 10.5.2.10000-5.
Then upgraded to 10.5.13900-12.
Didn't seem to be an issue then came in today and its broken.

The info you requested is pasted below

admin:file view activelog platform/log/diag1.log

03-02-2016 18:46:31                       Diagnostics Version: 1.0.0
03-02-2016 18:46:31                       getting hardware model
[/usr/local/bin/base_scripts/sd_hwdetect HWModel]
03-02-2016 18:46:32                       Hardware Model: VMware
03-02-2016 18:46:32                       getting verson number [rpm -q
--nodigest --nosignature master | sed -e "s/master-//"]
03-02-2016 18:46:32                       Version: 10.5.2
03-02-2016 18:46:33 disk_space:           Is valid module: True
03-02-2016 18:46:33 disk_files:           Is valid module: True
03-02-2016 18:46:33 service_manager:      Is valid module: True
03-02-2016 18:46:33 tomcat:               Is valid module: True
03-02-2016 18:46:33 tomcat_deadlocks:     Is valid module: True
03-02-2016 18:46:33 tomcat_keystore:      Is valid module: True
03-02-2016 18:46:33 tomcat_connectors:    Is valid module: True
03-02-2016 18:46:33 tomcat_threads:       Is valid module: True
03-02-2016 18:46:33 tomcat_memory:        Is valid module: True
03-02-2016 18:46:33 tomcat_sessions:      Is valid module: True
03-02-2016 18:46:33 tomcat_heapdump:      Is valid module: True
03-02-2016 18:46:33 validate_network:     Product specific XML file:
/usr/local/platform/conf/cli/cliProduct.xml
03-02-2016 18:46:33 validate_network:     val: true
03-02-2016 18:46:33 validate_network:     Is valid module: True
03-02-2016 18:46:33 validate_network_adv: Is valid module: False

options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) :


admin:
admin:file view activelog platform/log/diag1.log

03-02-2016 18:46:31                       Diagnostics Version: 1.0.0
03-02-2016 18:46:31                       getting hardware model
[/usr/local/bin/base_scripts/sd_hwdetect HWModel]
03-02-2016 18:46:32                       Hardware Model: VMware
03-02-2016 18:46:32                       getting verson number [rpm -q
--nodigest --nosignature master | sed -e "s/master-//"]
03-02-2016 18:46:32                       Version: 10.5.2
03-02-2016 18:46:33 disk_space:           Is valid module: True
03-02-2016 18:46:33 disk_files:           Is valid module: True
03-02-2016 18:46:33 service_manager:      Is valid module: True
03-02-2016 18:46:33 tomcat:               Is valid module: True
03-02-2016 18:46:33 tomcat_deadlocks:     Is valid module: True
03-02-2016 18:46:33 tomcat_keystore:      Is valid module: True
03-02-2016 18:46:33 tomcat_connectors:    Is valid module: True
03-02-2016 18:46:33 tomcat_threads:       Is valid module: True
03-02-2016 18:46:33 tomcat_memory:        Is valid module: True
03-02-2016 18:46:33 tomcat_sessions:      Is valid module: True
03-02-2016 18:46:33 tomcat_heapdump:      Is valid module: True
03-02-2016 18:46:33 validate_network:     Product specific XML file:
/usr/local/platform/conf/cli/cliProduct.xml
03-02-2016 18:46:33 validate_network:     val: true
03-02-2016 18:46:33 validate_network:     Is valid module: True
03-02-2016 18:46:33 validate_network_adv: Is valid module: False

options: q=quit, n=next, p=prev, b=begin, e=end (lines 1 - 20 of 54) :
03-02-2016 18:46:33 raid:                 getting cpu speed
[/usr/local/bin/base_scripts/sd_hwdetect CPUSpeed]
03-02-2016 18:46:33 raid:                 CPU Speed: 2500
03-02-2016 18:46:33 raid:                 model = VMware
03-02-2016 18:46:33 raid:                 Is valid module: True
03-02-2016 18:46:33 system_info:          Is valid module: True
03-02-2016 18:46:33 ntp_reachability:     Is valid module: True
03-02-2016 18:46:33 ntp_clock_drift:      Is valid module: True
03-02-2016 18:46:33 ntp_stratum:          Is valid module: True
03-02-2016 18:46:33 sdl_fragmentation:    Is valid module: True
03-02-2016 18:46:33 sdi_fragmentation:    Is valid module: True
03-02-2016 18:46:33 ipv6_networking:      IPV6INIT=no
03-02-2016 18:46:33 ipv6_networking:      IPv6 initialized: no
03-02-2016 18:46:33 ipv6_networking:      False
03-02-2016 18:46:33 ipv6_networking:      Is valid module: False
03-02-2016 18:46:33
03-02-2016 18:46:33                       --> executing test
[validate_network], fix: fixauto, stop on error: False
03-02-2016 18:46:33
03-02-2016 18:46:33 validate_network:     ------------------
03-02-2016 18:46:33 validate_network:     Testing networking, but skipping
duplicate IP test.
03-02-2016 18:46:33 validate_network:     checking network
[/usr/local/bin/base_scripts/validateNetworking.sh -n]

options: q=quit, n=next, p=prev, b=begin, e=end (lines 21 - 40 of 54) :
03-02-2016 18:46:33 validate_network:     retrieving pub name from
[/usr/local/platform/conf/platformConfig.xml]
03-02-2016 18:46:33 validate_network:     Hostname: [XXXXXXXX]
03-02-2016 18:46:33 validate_network:     found pub name [XXXXXXX]
03-02-2016 18:46:33 validate_network:     checking /etc/hosts [grep -q
`hostname` /etc/hosts]
03-02-2016 18:46:33 validate_network:     Finding cluster nodes
[/usr/local/bin/base_scripts/list_cluster.sh]
03-02-2016 18:46:33 validate_network:     running
[./diag_validate_network_sftp.exp sftpuser at xxx.xxx.9x.101>/dev/null]
03-02-2016 18:46:35 validate_network:     running
[./diag_validate_network_sftp.exp sftpuser at xxx.xxx.9x.102>/dev/null]
03-02-2016 18:46:46 validate_network:     running
[./diag_validate_network_sftp.exp sftpuser at xxx.xxx.4x.101>/dev/null]
03-02-2016 18:46:47 validate_network:     running
[./diag_validate_network_sftp.exp sftpuser at xxx.xxx.9x.130>/dev/null]
03-02-2016 18:46:48 validate_network:     does test script exist
[/usr/local/bin/base_scripts/networkDiagnostic.sh]
03-02-2016 18:46:48 validate_network:     test script exists
03-02-2016 18:46:48 validate_network:     run network script via expect
[./diag_validate_network.exp > /dev/null]
03-02-2016 18:46:48 validate_network:     result: 0, message: Passed


end of the file reached
options: q=quit, n=next, p=prev, b=begin, e=end (lines 41 - 54 of 54) :
admin:



admin:utils ntp status
ntpd (pid 8970) is running...

     remote           refid      st t when poll reach   delay   offset
jitter
==============================================================================
 xxx.xxx.55.203    .INIT.          16 u    - 1024    0    0.000    0.000
0.000
*xxx.xxx.5.203     LOCAL(1)         8 u  268  512  377    0.611    0.304
0.289


synchronised to NTP server (xxx.xxx.5.203) at stratum 9
   time correct to within 37 ms
   polling server every 512 s

Current time in UTC is : Wed Mar  2 18:49:09 UTC 2016
Current time in Europe/London is : Wed Mar  2 18:49:09 GMT 2016
admin:



admin:utils ntp server list
xxx.xxx.55.203

xxx.xxx.5.203
admin:


Regards

On 2 March 2016 at 17:36, Ryan Huff <ryanhuff at outlook.com> wrote:

> Not that I'm suggesting you not call TAC but the engineer in me just keeps
> going ....
>
> What (version) did you upgrade from and did you upgrade in-place VMs,
> DRS/Rebuild or P->V?
>
> Do you know if at any point post upgrade, the cluster was healthy and then
> failed or it has always been in a degraded state since the upgrade?
>
> Can you show me the output (from the publisher);
>
> - utils diagnose module validate_network
> - show ntp status
> - show ntp server list
>
> Thanks,
>
> Ryan
>
> > On Mar 2, 2016, at 12:25 PM, Ryan Huff <ryanhuff at outlook.com> wrote:
> >
> > I'd go through a quick checklist while calling in a severity 1 TAC case;
> >
> > - forward and reverse DNS for all cluster nodes (and resolving to the
> correct addresses)
> >
> > - verify the processNodes, if using hosts or fqdn, are correctly
> resolvable. This will prevent A Cisco DB from starting as well as GUI
> authentication
> >
> > - do not have an absurd clock sync on the nodes (Stratum 3 or better)
> >
> > Thanks,
> >
> > Ryan
> >
> >> On Mar 2, 2016, at 12:13 PM, Andy Carse <andy.carse at gmail.com> wrote:
> >>
> >> I thought I was home and dry with this upgrade, but it would seem that
> the gods have deserted me.
> >>
> >> I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything
> seemed ok.
> >> This morning I've come in to find that the database on the publisher
> won't start.
> >> So I've tried
> >> 1. reboot of the cluster (its not gone live yet) no change.
> >> 2. Utils service start A Cisco DB
> >> 2. tried dbreplication stop on the subs, then the publisher.
> >>           dbreplication dropddmindb on the subs
> >>           dbreplication dropadmindb on the pub
> >> The pub comes back with "DropAdminDB cannot be executed on standalone
> or Cores cluster"
> >>
> >> I can't even web to ccmadmin on the pub and I forgot to carry out the
> "Golden Rule" of taking a backup soon after the upgrade.
> >> If I try to RTM that also fails......
> >>
> >> Is it time for a start from scratch moment?
> >>
> >>
> >>
> >> --
> >> Rgds Andy
> >>
> >> _______________________________________________
> >> cisco-voip mailing list
> >> cisco-voip at puck.nether.net
> >> https://puck.nether.net/mailman/listinfo/cisco-voip
> > _______________________________________________
> > cisco-voip mailing list
> > cisco-voip at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-voip
>



-- 
Rgds Andy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20160302/2a6c5e5f/attachment.html>

More information about the cisco-voip mailing list