[cisco-voip] CUCM Upgrade woes

[Ryan Ratliff (rratliff)]

Not saying this is responsible for your failure but you need a Stratum 3 NTP or better. Your current one is stratum 9.

At any point did you change hostnames or IP address on this node?

-Ryan

On Mar 2, 2016, at 2:16 PM, Brian V <bvanbens at gmail.com<mailto:bvanbens at gmail.com>> wrote:

Noticing your upgrade version is 10.5.2.13900
could it be related to the new bug with vmware tools that fills the disk.

new sev2 bug with 9 TAC cases attached to it already
 VMware Tools 10.0 update fails on CUCM 10.5/11.0 with selinux denials
CSCux90747
Symptom:
VMware Tools upgrade fails due to various Selinux denials. VI-Client indicates tools status as Not running, Not Installed.

The following selinux denial is seen in System Logs (messages) when VMtools update attempt fails either via VI-client initiated automatic update or Automatic Update that takes place during boot up as long as VM Setting "Check and upgrade VMware Tools before each power on" is checked.

Feb 25 20:20:18 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/bin/perl from create access on the directory /var/lib/. For complete SELinux messages. run sealert -l 84003ecc-5de4-4e59-9ab8-1e7a28225c18

The following selinux denials is seen in System Logs (messages) when Vmtools update to 10.0 version or above is successful after putting System OS Security to Permissive mode followed by Update of Tools and then putting System OS Security back to Enforcing mode.

Feb 22 16:34:23 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the directory requests. For complete SELinux messages. run sealert -l 76069c58-d7be-482f-8391-4eb94d51ecd9
Feb 22 16:34:23 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the directory requests. For complete SELinux messages. run sealert -l 76069c58-d7be-482f-8391-4eb94d51ecd9
Feb 22 16:34:24 cucm-pub user 3 setroubleshoot: SELinux is preventing CThreadUtils::s from write access on the directory output. For complete SELinux messages. run sealert -l 9e71ec6f-cd83-43a5-8564-14f66e77e4ff
Feb 22 16:34:24 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the directory providerReg. For complete SELinux messages. run sealert -l 76069c58-d7be-482f-8391-4eb94d51ecd9
Feb 22 16:34:25 cucm-pub user 3 setroubleshoot: SELinux is preventing CThreadUtils::s from write access on the directory output. For complete SELinux messages. run sealert -l 9e71ec6f-cd83-43a5-8564-14f66e77e4ff

Under these conditions where VMtools 10.0 is running with CUCM 10.X or 11.X, Putting OS Security mode back to enforcing will inevitably lead to:

1. All available virtual memory is consumed by settroubleshootd because of continuous selinux denials
2. vmware-caf logs consume 100% of the active partition due to selinux denying log rotation (logs are in /usr/lib/vmware-caf/pme/bin).

Conditions:
Problem is seen after Upgrading to latest builds of ESXi 5.5 or 6.0 builds greater than 3248547 which bundles 10240 (10.0.0) version of VMware Tools and brings in a new vmware-caf functionality.

Workaround:
DO NOT UPDATE Vmware tools to version 10240 (10.0.0) or above if you are running CUCM 10.x or 11.X

If you have already attempted an earlier acceptable workaround to Update VMware tools to version 10.0 or above and restored OS Security mode to enforcing, you may observe a flooding of selinux denials in System messages logs.

Under these condition the System will run out memory due to excessive setroubleshootd logging and eventually the run out of Active Root Partition which may prevent further access to Platform CLI and/or ability to create Remote support account to recover from this condition.

!!! This is extremely important !!! If you must keep selinux in enforcing mode all the time due to security concerns, do NOT upgrade to ESXi 6.0 and/or attempt to update vmtools install

If you have already attempted an earlier workaround to Update VMware tools to version 10.0 or above Revert OS Security mode to permissive via (utils os secure permissive) immediately and contact TAC for recovery options.

Further Problem Description:
Put OS Security back to enforcing mode only if you are absolutely sure that you are Updating VMware Tools to a version below 10.0. For reference look at this VMware tools version mapping doc to correlate your ESXi Host builds to bundled vmtools versions.


On 3/2/2016 11:12 AM, Andy Carse wrote:
I thought I was home and dry with this upgrade, but it would seem that the gods have deserted me.

I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything seemed ok.
This morning I've come in to find that the database on the publisher won't start.
So I've tried
1. reboot of the cluster (its not gone live yet) no change.
2. Utils service start A Cisco DB
2. tried dbreplication stop on the subs, then the publisher.
           dbreplication dropddmindb on the subs
           dbreplication dropadmindb on the pub
The pub comes back with "DropAdminDB cannot be executed on standalone or Cores cluster"

I can't even web to ccmadmin on the pub and I forgot to carry out the "Golden Rule" of taking a backup soon after the upgrade.
If I try to RTM that also fails......

Is it time for a start from scratch moment?



--
Rgds Andy




_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip


_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20160302/cd48b5df/attachment.html>