[cisco-voip] CUCM Upgrade failure...

Jonathan Charles jonvoip at gmail.com
Tue Nov 17 13:00:27 EST 2015 

Well the battery looks fine:

[image: Inline image 1]

But the write cache is disabled...

[image: Inline image 2]


I wonder if that is causing the time out...



Jonathan



On Tue, Nov 17, 2015 at 11:58 AM, Ryan Huff <ryanhuff at outlook.com> wrote:

> Just as a note of practice; I wouldn't do anything to a running CCM at the
> OS level during production, at least not without proper maintenance
> declaration to the business unit [image: 😊]. "*A maintenance window a
> day, keeps the resumes away*" .... lol.
>
>
> If you've got RAID issues; I would get that solved before doing anything
> else. If DRF services are still working (50/50 chance) I would get a good
> DRS on that cluster *ASAP*. If DRS doesn't work, hopefully BAT still
> does, I would do a BAT export on everything ASAP.
>
>
> An upgrade would be the least of my worries at this point. He is a
> reference to the particular error message you show in the image.(
> http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/upgrade/9_1_1/CUCM_BK_UAEC4331_00_upgrade-guide-cucm-91/CUCM_BK_UAEC4331_00_upgrade-guide-cucm-91_chapter_0110.html#CUP0_RF_WD01249D_00
> ).
>
>
> -Ryan
>
> ------------------------------
> *From:* Jonathan Charles <jonvoip at gmail.com>
> *Sent:* Tuesday, November 17, 2015 12:45 PM
> *To:* Dave Goodwin
> *Cc:* Ryan Huff; cisco-voip at puck.nether.net
>
> *Subject:* Re: [cisco-voip] CUCM Upgrade failure...
>
> I did notice this on the Publisher OS Admin - Install/Upgrade page:
>
> [image: Inline image 1]
>
> I wonder if that's causing it...
>
>
>
> Jonathan
>
> On Tue, Nov 17, 2015 at 11:40 AM, Jonathan Charles <jonvoip at gmail.com>
> wrote:
>
>> Yep, didn't work... same error. Let me check the other nodes...
>>
>> On Tue, Nov 17, 2015 at 11:34 AM, Jonathan Charles <jonvoip at gmail.com>
>> wrote:
>>
>>> Logged into OS Admin on the pub, and it didn't give me the option to
>>> assume control... can I run discovery/migration during production? Just
>>> want to make sure I don't pop anything....
>>>
>>>
>>>
>>> Jonathan
>>>
>>> On Tue, Nov 17, 2015 at 11:30 AM, Dave Goodwin <
>>> dave.goodwin at december.net> wrote:
>>>
>>>> I had a similar symptom a couple months ago - discovery timed out on
>>>> one or more nodes. I found a supportforum post from another individual
>>>> having that issue, and it appeared that PCD got "stuck" installing
>>>> the ciscocm.ucmap_platformconfig.cop file in the middle. The way to work
>>>> around it was to log into the Software Install/Upgrade page on the affected
>>>> node(s) and you can see you'll have the option to Assume Control of a
>>>> currently running install. Do that, and click through to complete the
>>>> install. Then try to do the discovery again. Unknown if there was an
>>>> existing bug causing this problem, but the above workaround steps solved my
>>>> problem.
>>>>
>>>> On Tue, Nov 17, 2015 at 12:25 PM, Jonathan Charles <jonvoip at gmail.com>
>>>> wrote:
>>>>
>>>>> I just posted the log... I am not sure from reading it...
>>>>>
>>>>> discovery node 1617 failed with errorcode 1-3
>>>>>
>>>>>
>>>>> Jonathan
>>>>>
>>>>> On Tue, Nov 17, 2015 at 11:23 AM, Ryan Huff <ryanhuff at outlook.com>
>>>>> wrote:
>>>>>
>>>>>> Jonathan,
>>>>>>
>>>>>> What timesout on the publisher? Are you referencing when PCD tries to
>>>>>> do the cluster discovery on the existing cluster?
>>>>>>
>>>>>> -Ryan
>>>>>>
>>>>>>
>>>>>> Sent from my T-Mobile 4G LTE Device
>>>>>>
>>>>>>
>>>>>> -------- Original message --------
>>>>>> From: Jonathan Charles
>>>>>> Date:11/17/2015 12:06 PM (GMT-05:00)
>>>>>> To: Anthony Holloway
>>>>>> Cc: cisco-voip at puck.nether.net
>>>>>> Subject: Re: [cisco-voip] CUCM Upgrade failure...
>>>>>>
>>>>>> Nope, not all good... new error... the upgrade failed so I deleted
>>>>>> the cluster and re-added... it finds all of the subscribers, but it times
>>>>>> out on the publisher....
>>>>>>
>>>>>> I have verified all services are running on the Pub and it looks
>>>>>> clean...
>>>>>>
>>>>>>
>>>>>>
>>>>>> Jonathan
>>>>>>
>>>>>> On Mon, Nov 16, 2015 at 10:20 AM, Anthony Holloway <
>>>>>> avholloway+cisco-voip at gmail.com> wrote:
>>>>>>
>>>>>>> Looks like you're all good now, but as a heads up to everyone else,
>>>>>>> don't stop at checking NTP with "utils ntp status".  You will fail to
>>>>>>> upgrade if your NTP configuration has an FQHN for the NTP server which
>>>>>>> begins with a digit.
>>>>>>>
>>>>>>> E.g., 0.pool.ntp.org
>>>>>>>
>>>>>>> You will not see the hostname in the output of "utils ntp status",
>>>>>>> as it will only show you the resolved IP address.  So, you will also need
>>>>>>> to issue a "utils ntp config" to see what value was entered by the
>>>>>>> administrator.
>>>>>>>
>>>>>>> This is the only defect reference I found, though my upgrade I hit
>>>>>>> it on was an 8.6 to 10.5 Refresh Upgrade (RU) (Not PCD).
>>>>>>>
>>>>>>> https://tools.cisco.com/bugsearch/bug/CSCtj07817
>>>>>>>
>>>>>>> On Sun, Nov 15, 2015 at 10:43 PM, Jonathan Charles <
>>>>>>> jonvoip at gmail.com> wrote:
>>>>>>>
>>>>>>>> OK, a reboot of CPCD got it passed that error...
>>>>>>>>
>>>>>>>>
>>>>>>>> Jonathan
>>>>>>>>
>>>>>>>> On Sun, Nov 15, 2015 at 9:40 PM, Jonathan Charles <
>>>>>>>> jonvoip at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Yeah, the error I am getting is:
>>>>>>>>>
>>>>>>>>> 1 nodes(s) in Export task action ID #1127... on the Publisher...
>>>>>>>>>
>>>>>>>>> I will try rebooting everything...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jonathan
>>>>>>>>>
>>>>>>>>> On Sun, Nov 15, 2015 at 9:35 PM, Ryan Huff <ryanhuff at outlook.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Looks healthy ...
>>>>>>>>>>
>>>>>>>>>> I recall trying PCD once and I hit really strange issues too. For
>>>>>>>>>> that upgrade, I ultimately abandoned PCD and built new VMs with the Answer
>>>>>>>>>> File Generator then a DRS backup/restore.
>>>>>>>>>>
>>>>>>>>>> Not sure where you are in your timeline or if it is that
>>>>>>>>>> important but it is definitely something I would consider. Sometimes you
>>>>>>>>>> can spend more time trying to get the silly tools to work, than to just do
>>>>>>>>>> the work yourself.
>>>>>>>>>>
>>>>>>>>>> Google is littered with PCD weirdness; great idea of an
>>>>>>>>>> application, just not there yet IMO.
>>>>>>>>>>
>>>>>>>>>> -Ryan
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sent from my iPad
>>>>>>>>>> On Nov 15, 2015, at 10:19 PM, Jonathan Charles <jonvoip at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Everything looks good....
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> admin:utils ntp status
>>>>>>>>>> ntpd (pid 19674) is running...
>>>>>>>>>>
>>>>>>>>>>      remote           refid      st t when poll reach   delay
>>>>>>>>>> offset  jitter
>>>>>>>>>>
>>>>>>>>>> ==============================================================================
>>>>>>>>>>  127.127.1.0     .LOCL.          10 l   21   64  377    0.000
>>>>>>>>>>  0.000   0.001
>>>>>>>>>>  10.0.31.2       10.0.31.3        3 u  175 1024  377    0.635
>>>>>>>>>> -5.771   2.042
>>>>>>>>>> *10.0.31.3       129.6.15.29      2 u  970 1024  377    0.510
>>>>>>>>>>  -11.340   0.449
>>>>>>>>>>  10.1.31.2       10.0.31.3        3 u  490 1024  377    0.850
>>>>>>>>>> -9.114   4.881
>>>>>>>>>> +10.1.31.3       129.6.15.29      2 u  184 1024  377    0.817
>>>>>>>>>> -4.085   5.355
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> synchronised to NTP server (10.0.31.3) at stratum 3
>>>>>>>>>>    time correct to within 68 ms
>>>>>>>>>>    polling server every 1024 s
>>>>>>>>>>
>>>>>>>>>> Current time in UTC is : Mon Nov 16 03:16:14 UTC 2015
>>>>>>>>>> Current time in America/Chicago is : Sun Nov 15 21:16:14 CST 2015
>>>>>>>>>> admin:
>>>>>>>>>>
>>>>>>>>>> admin:utils diagnose module validate_network
>>>>>>>>>>
>>>>>>>>>> Log file: platform/log/diag1.log
>>>>>>>>>>
>>>>>>>>>> Starting diagnostic test(s)
>>>>>>>>>> ===========================
>>>>>>>>>> test - validate_network    : Passed
>>>>>>>>>>
>>>>>>>>>> Diagnostics Completed
>>>>>>>>>>
>>>>>>>>>> admin:#
>>>>>>>>>>
>>>>>>>>>> admin:utils dbreplication runtimestate
>>>>>>>>>>
>>>>>>>>>> DB and Replication Services: ALL RUNNING
>>>>>>>>>>
>>>>>>>>>> Cluster Replication State: Replication repair command started at:
>>>>>>>>>> 2014-06-20-23-22
>>>>>>>>>>      Replication repair command COMPLETED 541 tables processed
>>>>>>>>>> out of 541
>>>>>>>>>>      Errors or Mismatches Were Found:
>>>>>>>>>>
>>>>>>>>>>      Use 'file view activelog
>>>>>>>>>> cm/trace/dbl/sdi/ReplicationRepair.2014_06_20_23_22_51.out' to see the
>>>>>>>>>> details
>>>>>>>>>>
>>>>>>>>>> DB Version: ccm8_6_2_20000_2
>>>>>>>>>> Number of replicated tables: 541
>>>>>>>>>>
>>>>>>>>>> Cluster Detailed View from PUB (5 Servers):
>>>>>>>>>>
>>>>>>>>>>                                 PING            REPLICATION
>>>>>>>>>> REPL.   DBver&  REPL.       REPLICATION SETUP
>>>>>>>>>> SERVER-NAME     IP ADDRESS      (msec)  RPC?    STATUS
>>>>>>>>>>  QUEUE   TABLES  LOOP?       (RTMT) & details
>>>>>>>>>> -----------     ------------    ------  ----    -----------
>>>>>>>>>> -----   ------- -----       -----------------
>>>>>>>>>> IPTCMS02  10.0.126.12     0.196   Yes     Connected       0
>>>>>>>>>> match   Yes         (2) Setup Completed
>>>>>>>>>> IPTCMS01  10.0.126.11     0.151   Yes     Connected       0
>>>>>>>>>> match   Yes         (2) Setup Completed
>>>>>>>>>> IPTCMP    10.0.126.10     0.065   Yes     Connected       0
>>>>>>>>>> match   Yes         (2) PUB Setup Completed
>>>>>>>>>> IPTCMS03  10.1.126.13     0.545   Yes     Connected       0
>>>>>>>>>> match   Yes         (2) Setup Completed
>>>>>>>>>> IPTCMS04  10.1.126.14     0.527   Yes     Connected       0
>>>>>>>>>> match   Yes         (2) Setup Completed
>>>>>>>>>>
>>>>>>>>>> admin:
>>>>>>>>>>
>>>>>>>>>> On Sun, Nov 15, 2015 at 9:14 PM, Ryan Huff <ryanhuff at outlook.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Also worth noting that if CCM NTP is synchronized to a Windows
>>>>>>>>>>> server (even if it shows Stratum 3 or better); that is a problem you'll
>>>>>>>>>>> need to correct as SNTP can play hell with UCOS and do some pretty weird
>>>>>>>>>>> stuff.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Ryan
>>>>>>>>>>>
>>>>>>>>>>> On Nov 15, 2015, at 10:06 PM, Ryan Huff <ryanhuff at outlook.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> If the FROM CCM version was unrestricted, it would say "
>>>>>>>>>>> *Unrestricted*" after the version number on the "*Active Master
>>>>>>>>>>> Version"* line. If it does not say "*Unrestricted*", then it is
>>>>>>>>>>> the more common restricted version.
>>>>>>>>>>>
>>>>>>>>>>> As to your original issue, I would start with all the usual
>>>>>>>>>>> suspects. Is the FROM CCM cluster healthy to start with; dns, ntp,
>>>>>>>>>>> replication ... etc?
>>>>>>>>>>>
>>>>>>>>>>> From CCM:
>>>>>>>>>>>
>>>>>>>>>>> 1.) #utils diagnose module validate_network
>>>>>>>>>>>         (Should see *Passed*)
>>>>>>>>>>>
>>>>>>>>>>> 2.) #utils ntp status
>>>>>>>>>>>         (Pub should be synchronized and Strata 3 or better)
>>>>>>>>>>>
>>>>>>>>>>> 3.) #utils dbreplication runtimestate
>>>>>>>>>>>         (Should see *2 - Setup Completed* for all nodes)
>>>>>>>>>>>
>>>>>>>>>>> If PCD is moving the apps to a new platform/chassis, make sure
>>>>>>>>>>> the *target* environment can reach all the same network assets
>>>>>>>>>>> as the *from* environment.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Ryan
>>>>>>>>>>>
>>>>>>>>>>> On Nov 15, 2015, at 9:34 PM, Jonathan Charles <jonvoip at gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> It just says the version number...
>>>>>>>>>>>
>>>>>>>>>>> admin:show version active
>>>>>>>>>>> Active Master Version: 8.6.2.20000-2
>>>>>>>>>>> Active Version Installed Software Options:
>>>>>>>>>>> cmterm-7942_7962-sccp.9-3-1ES27-rel.cop
>>>>>>>>>>> cmterm-devicepack8.6.2.24118-1.cop
>>>>>>>>>>> ciscocm.refresh_upgrade_v1.1.cop
>>>>>>>>>>> ciscocm.ucmap_platformconfig.cop
>>>>>>>>>>> ciscocm.migrate-export-v1.12.cop
>>>>>>>>>>> admin:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Jonahan
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Nov 15, 2015 at 7:26 PM, Ryan Huff <ryanhuff at outlook.com
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Did you actually dump the logs to a serial interface (curious
>>>>>>>>>>>> what it shows)?
>>>>>>>>>>>>
>>>>>>>>>>>> On the FROM CCM, goto the CLI of pub (or sub) and do a "show
>>>>>>>>>>>> version active"; it will tell you if you have the UNREST.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>>
>>>>>>>>>>>> > On Nov 15, 2015, at 7:55 PM, Jonathan Charles <
>>>>>>>>>>>> jonvoip at gmail.com> wrote:
>>>>>>>>>>>> >
>>>>>>>>>>>> > Using PCD on CUCM 10.5.2.11901 got the following error:
>>>>>>>>>>>> >
>>>>>>>>>>>> > <image.png>
>>>>>>>>>>>> >
>>>>>>>>>>>> > It seems to imply I am not matching restricted vs.
>>>>>>>>>>>> unrestricted...
>>>>>>>>>>>> >
>>>>>>>>>>>> > Any easy way to find out?
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > Jonathan
>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>> > cisco-voip mailing list
>>>>>>>>>>>> > cisco-voip at puck.nether.net
>>>>>>>>>>>> > https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> cisco-voip mailing list
>>>>>>>>>>> cisco-voip at puck.nether.net
>>>>>>>>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> cisco-voip mailing list
>>>>>>>> cisco-voip at puck.nether.net
>>>>>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> cisco-voip mailing list
>>>>> cisco-voip at puck.nether.net
>>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/7eb7c186/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 42074 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/7eb7c186/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OutlookEmoji-?.png
Type: image/png
Size: 488 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/7eb7c186/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 16236 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/7eb7c186/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 12690 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/7eb7c186/attachment-0003.png>

More information about the cisco-voip mailing list