[cisco-voip] high I/O Wait on one core

[Haas, Neal]

What version are you? 7 or 8


Neal Haas

From: cisco-voip-bounces at puck.nether.net [mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Wes Sisk
Sent: Friday, April 05, 2013 9:09 AM
To: Erick Wellnitz
Cc: cisco-voip
Subject: Re: [cisco-voip] high I/O Wait on one core

cmoninit is one of the informix database processes. There are are several processes that each do different roles. it's not simple to identify the exact role of a specific oninit process.

<quote>
Looks like each of the oninit processes implements a different (or parallel or redundant) part of the overall database server:

-bash-3.2$ onstat -g sch

IBM Informix Dynamic Server Version 11.50.UC8X6   -- On-Line -- Up 3 days 16:22:33 -- 230984 Kbytes

VP Scheduler Statistics:
vp    pid       class       semops    busy waits  spins/wait
1     24024     cpu         15202292  0           15205721
2     24066     adm         0         0           0
3     24067     LIC         18        0           18
4     24068     DBFNC       1         0           1
5     24069     lio         136050    0           0
6     24070     pio         1287      0           0
7     24071     aio         903428    0           0
...

http://publib.boulder.ibm.com/infocenter/idshelp/v117/index.jsp
</quote>

Generally, yes, you're looking at something in the database generating lots of churn. THis could be CDRs, it could be CAR, it could also be AXL or high rate of change notifications.

Looks like you would benefit from something like this:
CLI show tech activesql

from:
CSCsz67357    Need an Informix profiler built into CLI

Regards,
Wes

On Apr 5, 2013, at 11:20 AM, Erick Wellnitz <ewellnitzvoip at gmail.com<mailto:ewellnitzvoip at gmail.com>> wrote:

caroninit seems to be the biggest offender (by about 50x) in both disk writes and cpu usage.  Am I correct in assuming this has something to do with call detail records?

On Fri, Apr 5, 2013 at 9:02 AM, Tom Piscitell (tpiscite) <tpiscite at cisco.com<mailto:tpiscite at cisco.com>> wrote:
Erick,

You can use the FIOR utility from the CLI to identify which processes are writing to the disk.

admin:utils fior
      utils fior disable
      utils fior enable
      utils fior list
      utils fior start
      utils fior status
      utils fior stop
      utils fior top

Here is a typical use case:

1. Enable the FIOR utility before/during a time of High IO Wait
        admin:utils fior enable
        File I/O Statistics has been enabled.
        admin:utils fior start
        Loading fiostats module: ok
        Enabling fiostats : ok
        File I/O Statistics has been started.

2. Wait a couple minutes. FIOR will poll for data every 5 seconds I believe. Then use utils fior top to see whats hitting the CPU the hardest:

admin:utils fior top ?
Syntax:
utils fior top n sort_by [start=date-time] [stop=date-time]

         n:            number of processes
         sort_by:      read, write, read-rate, write-rate
         date-time:    of the form %H:%M, %H:%M:%S
                                   %a,%H:%M, %a,%H:%M:%S
                                   %Y-%m-%d,%H:%M, %Y-%m-%d %H:%M:%S
Example:
admin:utils fior top 10 write start=2010-04-20 10:00:00 stop=2010-04-20 10:30:00

This of course won't tell you *why* a process is hitting the disk, but it will at least show you who has the most read/writes. To answer the why question you would need to look at traces for the offending process/service.

HTH,
-Tom

On Apr 4, 2013, at 5:43 PM, Erick Wellnitz <ewellnitzvoip at gmail.com<mailto:ewellnitzvoip at gmail.com>> wrote:

> Hello all!
>
> I have a dual 4 core IBM 7835I3 which is my publisher.   One one core of the first CPU the I/O Wait is through the roof.  RTMT shows that writes to the hard drives are at between 600 and 700 MB/s which is exponentially higher than the subscriber on the same model of hardware.
>
> Short of calling TAC is there any way to figure out what is causing the extremely high volume of writes to the drives?  I already stopped most traces and looking at the processes doesn't give any clues.
>
> Thanks again!
>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20130405/291ce2b1/attachment.html>