[cisco-voip] CUCM Upgrade failure...

Tue Nov 17 12:58:08 EST 2015

Just as a note of practice; I wouldn't do anything to a running CCM at the OS level during production, at least not without proper maintenance declaration to the business unit [😊] . "A maintenance window a day, keeps the resumes away" .... lol.

If you've got RAID issues; I would get that solved before doing anything else. If DRF services are still working (50/50 chance) I would get a good DRS on that cluster ASAP. If DRS doesn't work, hopefully BAT still does, I would do a BAT export on everything ASAP.

An upgrade would be the least of my worries at this point. He is a reference to the particular error message you show in the image.(http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/upgrade/9_1_1/CUCM_BK_UAEC4331_00_upgrade-guide-cucm-91/CUCM_BK_UAEC4331_00_upgrade-guide-cucm-91_chapter_0110.html#CUP0_RF_WD01249D_00).

-Ryan

________________________________
From: Jonathan Charles <jonvoip at gmail.com>
Sent: Tuesday, November 17, 2015 12:45 PM
To: Dave Goodwin
Cc: Ryan Huff; cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] CUCM Upgrade failure...

I did notice this on the Publisher OS Admin - Install/Upgrade page:

[Inline image 1]

I wonder if that's causing it...

Jonathan

On Tue, Nov 17, 2015 at 11:40 AM, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:
Yep, didn't work... same error. Let me check the other nodes...

On Tue, Nov 17, 2015 at 11:34 AM, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:
Logged into OS Admin on the pub, and it didn't give me the option to assume control... can I run discovery/migration during production? Just want to make sure I don't pop anything....

Jonathan

On Tue, Nov 17, 2015 at 11:30 AM, Dave Goodwin <dave.goodwin at december.net<mailto:dave.goodwin at december.net>> wrote:
I had a similar symptom a couple months ago - discovery timed out on one or more nodes. I found a supportforum post from another individual having that issue, and it appeared that PCD got "stuck" installing the ciscocm.ucmap_platformconfig.cop file in the middle. The way to work around it was to log into the Software Install/Upgrade page on the affected node(s) and you can see you'll have the option to Assume Control of a currently running install. Do that, and click through to complete the install. Then try to do the discovery again. Unknown if there was an existing bug causing this problem, but the above workaround steps solved my problem.

On Tue, Nov 17, 2015 at 12:25 PM, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:
I just posted the log... I am not sure from reading it...

discovery node 1617 failed with errorcode 1-3

Jonathan

On Tue, Nov 17, 2015 at 11:23 AM, Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>> wrote:
Jonathan,

What timesout on the publisher? Are you referencing when PCD tries to do the cluster discovery on the existing cluster?

-Ryan

Sent from my T-Mobile 4G LTE Device

-------- Original message --------
From: Jonathan Charles
Date:11/17/2015 12:06 PM (GMT-05:00)
To: Anthony Holloway
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] CUCM Upgrade failure...

Nope, not all good... new error... the upgrade failed so I deleted the cluster and re-added... it finds all of the subscribers, but it times out on the publisher....

I have verified all services are running on the Pub and it looks clean...

Jonathan

On Mon, Nov 16, 2015 at 10:20 AM, Anthony Holloway <avholloway+cisco-voip at gmail.com<mailto:avholloway+cisco-voip at gmail.com>> wrote:
Looks like you're all good now, but as a heads up to everyone else, don't stop at checking NTP with "utils ntp status".  You will fail to upgrade if your NTP configuration has an FQHN for the NTP server which begins with a digit.

E.g., 0.pool.ntp.org<http://0.pool.ntp.org>

You will not see the hostname in the output of "utils ntp status", as it will only show you the resolved IP address.  So, you will also need to issue a "utils ntp config" to see what value was entered by the administrator.

This is the only defect reference I found, though my upgrade I hit it on was an 8.6 to 10.5 Refresh Upgrade (RU) (Not PCD).

https://tools.cisco.com/bugsearch/bug/CSCtj07817

On Sun, Nov 15, 2015 at 10:43 PM, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:
OK, a reboot of CPCD got it passed that error...

Jonathan

On Sun, Nov 15, 2015 at 9:40 PM, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:
Yeah, the error I am getting is:

1 nodes(s) in Export task action ID #1127... on the Publisher...

I will try rebooting everything...

Jonathan

On Sun, Nov 15, 2015 at 9:35 PM, Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>> wrote:
Looks healthy ...

I recall trying PCD once and I hit really strange issues too. For that upgrade, I ultimately abandoned PCD and built new VMs with the Answer File Generator then a DRS backup/restore.

Not sure where you are in your timeline or if it is that important but it is definitely something I would consider. Sometimes you can spend more time trying to get the silly tools to work, than to just do the work yourself.

Google is littered with PCD weirdness; great idea of an application, just not there yet IMO.

-Ryan

Sent from my iPad
On Nov 15, 2015, at 10:19 PM, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:

Everything looks good....

admin:utils ntp status
ntpd (pid 19674) is running...

     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 127.127.1.0     .LOCL.          10 l   21   64  377    0.000    0.000   0.001
 10.0.31.2       10.0.31.3        3 u  175 1024  377    0.635   -5.771   2.042
*10.0.31.3       129.6.15.29      2 u  970 1024  377    0.510  -11.340   0.449
 10.1.31.2       10.0.31.3        3 u  490 1024  377    0.850   -9.114   4.881
+10.1.31.3       129.6.15.29      2 u  184 1024  377    0.817   -4.085   5.355

synchronised to NTP server (10.0.31.3) at stratum 3
   time correct to within 68 ms
   polling server every 1024 s

Current time in UTC is : Mon Nov 16 03:16:14 UTC 2015
Current time in America/Chicago is : Sun Nov 15 21:16:14 CST 2015
admin:

admin:utils diagnose module validate_network

Log file: platform/log/diag1.log

Starting diagnostic test(s)
===========================
test - validate_network    : Passed

Diagnostics Completed

admin:#

admin:utils dbreplication runtimestate

DB and Replication Services: ALL RUNNING

Cluster Replication State: Replication repair command started at: 2014-06-20-23-22
     Replication repair command COMPLETED 541 tables processed out of 541
     Errors or Mismatches Were Found:

     Use 'file view activelog cm/trace/dbl/sdi/ReplicationRepair.2014_06_20_23_22_51.out' to see the details

DB Version: ccm8_6_2_20000_2
Number of replicated tables: 541

Cluster Detailed View from PUB (5 Servers):

                                PING            REPLICATION     REPL.   DBver&  REPL.       REPLICATION SETUP
SERVER-NAME     IP ADDRESS      (msec)  RPC?    STATUS          QUEUE   TABLES  LOOP?       (RTMT) & details
-----------     ------------    ------  ----    -----------     -----   ------- -----       -----------------
IPTCMS02  10.0.126.12     0.196   Yes     Connected       0       match   Yes         (2) Setup Completed
IPTCMS01  10.0.126.11     0.151   Yes     Connected       0       match   Yes         (2) Setup Completed
IPTCMP    10.0.126.10     0.065   Yes     Connected       0       match   Yes         (2) PUB Setup Completed
IPTCMS03  10.1.126.13     0.545   Yes     Connected       0       match   Yes         (2) Setup Completed
IPTCMS04  10.1.126.14     0.527   Yes     Connected       0       match   Yes         (2) Setup Completed

admin:

On Sun, Nov 15, 2015 at 9:14 PM, Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>> wrote:
Also worth noting that if CCM NTP is synchronized to a Windows server (even if it shows Stratum 3 or better); that is a problem you'll need to correct as SNTP can play hell with UCOS and do some pretty weird stuff.

Thanks,

Ryan

On Nov 15, 2015, at 10:06 PM, Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>> wrote:

If the FROM CCM version was unrestricted, it would say "Unrestricted" after the version number on the "Active Master Version" line. If it does not say "Unrestricted", then it is the more common restricted version.

As to your original issue, I would start with all the usual suspects. Is the FROM CCM cluster healthy to start with; dns, ntp, replication ... etc?

From CCM:

1.) #utils diagnose module validate_network
        (Should see Passed)

2.) #utils ntp status
        (Pub should be synchronized and Strata 3 or better)

3.) #utils dbreplication runtimestate
        (Should see 2 - Setup Completed for all nodes)

If PCD is moving the apps to a new platform/chassis, make sure the target environment can reach all the same network assets as the from environment.

Thanks,

Ryan

On Nov 15, 2015, at 9:34 PM, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:

It just says the version number...

admin:show version active
Active Master Version: 8.6.2.20000-2
Active Version Installed Software Options:
cmterm-7942_7962-sccp.9-3-1ES27-rel.cop
cmterm-devicepack8.6.2.24118-1.cop
ciscocm.refresh_upgrade_v1.1.cop
ciscocm.ucmap_platformconfig.cop
ciscocm.migrate-export-v1.12.cop
admin:

Jonahan

On Sun, Nov 15, 2015 at 7:26 PM, Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>> wrote:
Did you actually dump the logs to a serial interface (curious what it shows)?

On the FROM CCM, goto the CLI of pub (or sub) and do a "show version active"; it will tell you if you have the UNREST.

Thanks,

Ryan

> On Nov 15, 2015, at 7:55 PM, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:
>
> Using PCD on CUCM 10.5.2.11901 got the following error:
>
> <image.png>
>
> It seems to imply I am not matching restricted vs. unrestricted...
>
> Any easy way to find out?
>
>
>
> Jonathan
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/985eb5e5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 16236 bytes
Desc: image.png
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/985eb5e5/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OutlookEmoji-?.png
Type: image/png
Size: 488 bytes
Desc: OutlookEmoji-?.png
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/985eb5e5/attachment-0001.png>