[cisco-voip] CUCM Upgrade woes

[Brian V]

Noticing your upgrade version is 10.5.2.13900
could it be related to the new bug with vmware tools that fills the disk.

new sev2 bug with 9 TAC cases attached to it already

VMware Tools 10.0 update fails on CUCM 10.5/11.0 with selinux denials

CSCux90747

*Symptom:*

VMware Tools upgrade fails due to various Selinux denials. VI-Client 
indicates tools status as Not running, Not Installed.

The following selinux denial is seen in System Logs (messages) when 
VMtools update attempt fails either via VI-client initiated automatic 
update or Automatic Update that takes place during boot up as long as VM 
Setting "Check and upgrade VMware Tools before each power on" is checked.

Feb 25 20:20:18 cucm-pub user 3 setroubleshoot: SELinux is preventing 
/usr/bin/perl from create access on the directory /var/lib/. For 
complete SELinux messages. run sealert -l 
84003ecc-5de4-4e59-9ab8-1e7a28225c18

The following selinux denials is seen in System Logs (messages) when 
Vmtools update to 10.0 version or above is successful after putting 
System OS Security to Permissive mode followed by Update of Tools and 
then putting System OS Security back to Enforcing mode.

Feb 22 16:34:23 cucm-pub user 3 setroubleshoot: SELinux is preventing 
/usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the 
directory requests. For complete SELinux messages. run sealert -l 
76069c58-d7be-482f-8391-4eb94d51ecd9
Feb 22 16:34:23 cucm-pub user 3 setroubleshoot: SELinux is preventing 
/usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the 
directory requests. For complete SELinux messages. run sealert -l 
76069c58-d7be-482f-8391-4eb94d51ecd9
Feb 22 16:34:24 cucm-pub user 3 setroubleshoot: SELinux is preventing 
CThreadUtils::s from write access on the directory output. For complete 
SELinux messages. run sealert -l 9e71ec6f-cd83-43a5-8564-14f66e77e4ff
Feb 22 16:34:24 cucm-pub user 3 setroubleshoot: SELinux is preventing 
/usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the 
directory providerReg. For complete SELinux messages. run sealert -l 
76069c58-d7be-482f-8391-4eb94d51ecd9
Feb 22 16:34:25 cucm-pub user 3 setroubleshoot: SELinux is preventing 
CThreadUtils::s from write access on the directory output. For complete 
SELinux messages. run sealert -l 9e71ec6f-cd83-43a5-8564-14f66e77e4ff

Under these conditions where VMtools 10.0 is running with CUCM 10.X or 
11.X, Putting OS Security mode back to enforcing will inevitably lead to:

1. All available virtual memory is consumed by settroubleshootd because 
of continuous selinux denials
2. vmware-caf logs consume 100% of the active partition due to selinux 
denying log rotation (logs are in /usr/lib/vmware-caf/pme/bin).

*Conditions:*
Problem is seen after Upgrading to latest builds of ESXi 5.5 or 6.0 
builds greater than 3248547 which bundles 10240 (10.0.0) version of 
VMware Tools and brings in a new vmware-caf functionality.

*Workaround:*
DO NOT UPDATE Vmware tools to version 10240 (10.0.0) or above if you are 
running CUCM 10.x or 11.X

If you have already attempted an earlier acceptable workaround to Update 
VMware tools to version 10.0 or above and restored OS Security mode to 
enforcing, you may observe a flooding of selinux denials in System 
messages logs.

Under these condition the System will run out memory due to excessive 
setroubleshootd logging and eventually the run out of Active Root 
Partition which may prevent further access to Platform CLI and/or 
ability to create Remote support account to recover from this condition.

!!! This is extremely important !!! If you must keep selinux in 
enforcing mode all the time due to security concerns, do NOT upgrade to 
ESXi 6.0 and/or attempt to update vmtools install

If you have already attempted an earlier workaround to Update VMware 
tools to version 10.0 or above Revert OS Security mode to permissive via 
(utils os secure permissive) immediately and contact TAC for recovery 
options.

*Further Problem Description:*
Put OS Security back to enforcing mode only if you are absolutely sure 
that you are Updating VMware Tools to a version below 10.0. For 
reference look at this VMware tools version mapping doc to correlate 
your ESXi Host builds to bundled vmtools versions.



On 3/2/2016 11:12 AM, Andy Carse wrote:
> I thought I was home and dry with this upgrade, but it would seem that 
> the gods have deserted me.
>
> I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything 
> seemed ok.
> This morning I've come in to find that the database on the publisher 
> won't start.
> So I've tried
> 1. reboot of the cluster (its not gone live yet) no change.
> 2. Utils service start A Cisco DB
> 2. tried dbreplication stop on the subs, then the publisher.
>            dbreplication dropddmindb on the subs
>            dbreplication dropadmindb on the pub
> The pub comes back with "DropAdminDB cannot be executed on standalone 
> or Cores cluster"
>
> I can't even web to ccmadmin on the pub and I forgot to carry out the 
> "Golden Rule" of taking a backup soon after the upgrade.
> If I try to RTM that also fails......
>
> Is it time for a start from scratch moment?
>
>
>
> -- 
> Rgds Andy
>
>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20160302/5c32d91c/attachment.html>