[cisco-voip] CUCM Upgrade woes

Andy andy.carse at gmail.com
Thu Mar 3 14:27:54 EST 2016 

Ok, Just to bring this to a close I have instigated a BOLO for my database as it isn’t anywhere to be found on the publisher.

Rebuild time…….

Thanks for the responses and suggestions but even the database gods with their vast amounts of “pixie dust” and root access of course couldn’t find it.


Andy
andy.carse at gmail.com



> On 2 Mar 2016, at 21:52, Ryan Ratliff (rratliff) <rratliff at cisco.com> wrote:
> 
> You’ll notice the vmtools upgrade will fail if you upgrade ESXi to a version that includes the 10240 build (https://packages.vmware.com/tools/versions <https://packages.vmware.com/tools/versions>).
> Let it ride on the older vmtools (that won’t kill your system) until a COP file is out to fix the selinux policy.
> 
> -Ryan
> 
> On Mar 2, 2016, at 2:42 PM, Andy Carse <andy.carse at gmail.com <mailto:andy.carse at gmail.com>> wrote:
> 
> Esxi version is 5.5.0 Build 2068190 so that doesn't match.
> I don't remember it upgrading vmtools during the upgrade and the in the events it says upgrade vmtools which I'll refrain from at the moment.
> From the cli it only has utils vmtools refresh which is not very helpful if your trying to work out what version is currently running.
> 
> 
> On 2 March 2016 at 19:16, Brian V <bvanbens at gmail.com <mailto:bvanbens at gmail.com>> wrote:
> Noticing your upgrade version is 10.5.2.13900
> could it be related to the new bug with vmware tools that fills the disk.
> 
> new sev2 bug with 9 TAC cases attached to it already 
>  VMware Tools 10.0 update fails on CUCM 10.5/11.0 with selinux denials
> 
> CSCux90747
> 
> Symptom:
> VMware Tools upgrade fails due to various Selinux denials. VI-Client indicates tools status as Not running, Not Installed.
> 
> The following selinux denial is seen in System Logs (messages) when VMtools update attempt fails either via VI-client initiated automatic update or Automatic Update that takes place during boot up as long as VM Setting "Check and upgrade VMware Tools before each power on" is checked.
> 
> Feb 25 20:20:18 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/bin/perl from create access on the directory /var/lib/. For complete SELinux messages. run sealert -l 84003ecc-5de4-4e59-9ab8-1e7a28225c18
> 
> The following selinux denials is seen in System Logs (messages) when Vmtools update to 10.0 version or above is successful after putting System OS Security to Permissive mode followed by Update of Tools and then putting System OS Security back to Enforcing mode.
> 
> Feb 22 16:34:23 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the directory requests. For complete SELinux messages. run sealert -l 76069c58-d7be-482f-8391-4eb94d51ecd9
> Feb 22 16:34:23 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the directory requests. For complete SELinux messages. run sealert -l 76069c58-d7be-482f-8391-4eb94d51ecd9
> Feb 22 16:34:24 cucm-pub user 3 setroubleshoot: SELinux is preventing CThreadUtils::s from write access on the directory output. For complete SELinux messages. run sealert -l 9e71ec6f-cd83-43a5-8564-14f66e77e4ff
> Feb 22 16:34:24 cucm-pub user 3 setroubleshoot: SELinux is preventing /usr/lib/vmware-caf/pme/bin/ManagementAgentHost from read access on the directory providerReg. For complete SELinux messages. run sealert -l 76069c58-d7be-482f-8391-4eb94d51ecd9
> Feb 22 16:34:25 cucm-pub user 3 setroubleshoot: SELinux is preventing CThreadUtils::s from write access on the directory output. For complete SELinux messages. run sealert -l 9e71ec6f-cd83-43a5-8564-14f66e77e4ff
> 
> Under these conditions where VMtools 10.0 is running with CUCM 10.X or 11.X, Putting OS Security mode back to enforcing will inevitably lead to:
> 
> 1. All available virtual memory is consumed by settroubleshootd because of continuous selinux denials
> 2. vmware-caf logs consume 100% of the active partition due to selinux denying log rotation (logs are in /usr/lib/vmware-caf/pme/bin).
> 
> Conditions:
> Problem is seen after Upgrading to latest builds of ESXi 5.5 or 6.0 builds greater than 3248547 which bundles 10240 (10.0.0) version of VMware Tools and brings in a new vmware-caf functionality.
> 
> Workaround:
> DO NOT UPDATE Vmware tools to version 10240 (10.0.0) or above if you are running CUCM 10.x or 11.X
> 
> If you have already attempted an earlier acceptable workaround to Update VMware tools to version 10.0 or above and restored OS Security mode to enforcing, you may observe a flooding of selinux denials in System messages logs.
> 
> Under these condition the System will run out memory due to excessive setroubleshootd logging and eventually the run out of Active Root Partition which may prevent further access to Platform CLI and/or ability to create Remote support account to recover from this condition.
> 
> !!! This is extremely important !!! If you must keep selinux in enforcing mode all the time due to security concerns, do NOT upgrade to ESXi 6.0 and/or attempt to update vmtools install
> 
> If you have already attempted an earlier workaround to Update VMware tools to version 10.0 or above Revert OS Security mode to permissive via (utils os secure permissive) immediately and contact TAC for recovery options.
> 
> Further Problem Description:
> Put OS Security back to enforcing mode only if you are absolutely sure that you are Updating VMware Tools to a version below 10.0. For reference look at this VMware tools version mapping doc to correlate your ESXi Host builds to bundled vmtools versions.
> 
> 
> 
> On 3/2/2016 11:12 AM, Andy Carse wrote:
>> I thought I was home and dry with this upgrade, but it would seem that the gods have deserted me.
>> 
>> I upgraded to 10.5.2.13900-12 after some issue with GBNP, everything seemed ok.
>> This morning I've come in to find that the database on the publisher won't start.
>> So I've tried 
>> 1. reboot of the cluster (its not gone live yet) no change.
>> 2. Utils service start A Cisco DB
>> 2. tried dbreplication stop on the subs, then the publisher.
>>            dbreplication dropddmindb on the subs
>>            dbreplication dropadmindb on the pub
>> The pub comes back with "DropAdminDB cannot be executed on standalone or Cores cluster"
>>           
>> I can't even web to ccmadmin on the pub and I forgot to carry out the "Golden Rule" of taking a backup soon after the upgrade.
>> If I try to RTM that also fails......
>> 
>> Is it time for a start from scratch moment?
>> 
>> 
>> 
>> -- 
>> Rgds Andy
>> 
>> 
>> 
>>  _______________________________________________
>> cisco-voip mailing list
>> cisco-voip at puck.nether.net <mailto:cisco-voip at puck.nether.net>
>> https://puck.nether.net/mailman/listinfo/cisco-voip <https://puck.nether.net/mailman/listinfo/cisco-voip>
> 
> 
> 
> 
> -- 
> Rgds Andy
> 
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net <mailto:cisco-voip at puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-voip <https://puck.nether.net/mailman/listinfo/cisco-voip>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20160303/38c16bc2/attachment.html>

More information about the cisco-voip mailing list