[cisco-voip] CUCM Pub HWM Disk usage

Tue May 8 09:41:58 EDT 2012

The way the filesystem recovery works is to truncate corrupt files.  This can lead to system stability issues (depending on how important the truncated file was).  We have no way of repairing system files like this so the backup/rebuild/restore is the only option.

-Ryan

On May 4, 2012, at 11:24 AM, george.hendrix at l-3com.com wrote:

All,

  I ended up doing a cold reboot of the server (publisher), booted from the recovery cd and selected repair the file system option. The file system is good now and I am not having issues and able to do backups.  However, I noticed on the page that has instructions for repairing the file system highly recommends reimaging the system.  Do I really need to rebuild the system?

http://www.cisco.com/en/US/products/sw/voicesw/ps556/products_tech_note09186a0080b1f305.shtml

Note:  It is highly recommended that you perform the full system backup, and then re-image the system using DRS, to make sure the file system is stable in the future.

Bill

From: Ryan Ratliff [mailto:rratliff at cisco.com] 
Sent: Wednesday, May 02, 2012 2:28 PM
To: Hendrix, George (Bill) @ LSG - STRATIS
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] CUCM Pub HWM Disk usage

Something is a bit odd here, you should have some logs in those folders, unless I just flat gave you the wrong path.

Looking back at your logging partition it looks like the numbers are a bit off.  It appear to say you have more free space than the size of the partition.
Disk/logging        65574668K          68069024K                         K (101%)

Note that the 'used' column is empty, and I'd guess the 101% is a byproduct of some negative value not being accounted for.

I think it's time for a boot off the recovery cd to check the filesystem.   Got a backup?

-Ryan

On May 2, 2012, at 1:36 PM, george.hendrix at l-3com.com wrote:

Yeah, this was a fresh build.

I will try to look around…below is the output from the last commands you gave me.

admin:file list activelog tomcat/logs/* detail
dir count = 0, file count = 0
admin:file list activelog cm/log/informix/* detail
dir count = 0, file count = 0

I would imagine a cold reboot of the server probably wouldn’t help right?  Or worse, it won’t come back up.

Bill Hendrix 

From: Ryan Ratliff [mailto:rratliff at cisco.com] 
Sent: Wednesday, May 02, 2012 1:14 PM
To: Hendrix, George (Bill) @ LSG - STRATIS
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] CUCM Pub HWM Disk usage

The size of your inactive partition tells me you've never done an upgrade on this system so that's out. 

You can dig around and try to find a log somewhere under activelog to delete, I'd take a look at informix ccm.log and tomcat catalina.out files since I think there've been bugs for both of those in the past that made them get very large.

file list activelog tomcat/logs/* detail
file list activelog cm/log/informix/* detail

-Ryan

On May 2, 2012, at 11:19 AM, george.hendrix at l-3com.com wrote:

Ryan,

  Below is the output from the show status command showing the partition that’s full is the logging partition.

CPU Idle:  94.00%  System:  02.00%    User:  01.00%
  IOWAIT:  03.00%     IRQ:  00.00%    Soft:  00.00%   Intr/sec: 1019.00

Memory Total:        2053864K
        Free:             64692K
        Used:            1989172K
      Cached:         672404K
      Shared:              0K
     Buffers:           101304K

                                 Total                     Free                        Used
Disk/active         27632244K           15947620K           11403892K (42%)
Disk/inactive       27632272K          26195768K          32828K (1%)
Disk/logging        65574668K          68069024K                         K (101%)

When I tried to enter the file list command, below is the output.  So I was unable to find a file to delete with this command.

admin:file list inactivelog cm/trace/ccm/sdi/*
no such file or directory can be found

I also enter this command and got the response shown.
admin:file list activelog cm/trace/ccm/sdi/*
dir count = 0, file count = 0

Thanks,

Bill Hendrix  |  Network/VOIP Engineer
L3 STRATIS  POWERED BY EXCELLENCE

From: Ryan Ratliff [mailto:rratliff at cisco.com] 
Sent: Wednesday, May 02, 2012 9:55 AM
To: Hendrix, George (Bill) @ LSG - STRATIS
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] CUCM Pub HWM Disk usage

If the filesystem is truly readonly then you've no option but to reboot to recover it.   There is a distinction between the disk being full and being marked readonly by the OS however so make sure you know which one you are hitting.  

Can you paste the output of 'show status' to confirm which partition is full?  

Now to test the readonly filesystem then try and delete a file from the CLI.
file list inactivelog cm/trace/ccm/sdi/*
file delete inactivelog cm/trace/ccm/sdi/<insert filename here from above command output>

This will try to delete one of the ccm sdi traces from your inactive partition.  This will be completely harmless and if it works then your disk is just full, not readonly.

-Ryan

On May 2, 2012, at 7:58 AM, george.hendrix at l-3com.com wrote:

Hey Guys,

  I have a CUCM 6.1 Cluster that started sending me the following alert:

LogPartitionHighWaterMarkExceeded UsedDiskSpace : 101 MessageString : Disk utilization hits HWM!! Purging files...

Both the CDR Repository and CallManager Service are affected by this.  I can start them, but then they just stop within a short time.  I read somewhere to change the LWM and HWM to low numbers and tried that, but the disk usage is still staying at 101% (not sure how that is even possible).  I tried searching in RTMT for all logs on the Pub and it comes back that nothing is found.  I tried this by a date range and also within the last 60 days, nothing.  I also tried rebooting the server via command line and  received an error that the appliance restart failed.  From what I’ve read in various threads, the system seems to be in a read-only state and is not able to purge the files now, nor reboot.

Appreciate info anyone can provide as to how to clear the logs in the log partition.

Thanks,
Bill Hendrix

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20120508/4bf5d514/attachment.html>