[cisco-voip] high I/O Wait on one core

[Erick Wellnitz]

I reduced the retention duration from 60 to 32 it seems that made no
difference.
show tech activesql did not show anything.


On Fri, Apr 5, 2013 at 11:08 AM, Wes Sisk <wsisk at cisco.com> wrote:

> cmoninit is one of the informix database processes. There are are several
> processes that each do different roles. it's not simple to identify the
> exact role of a specific oninit process.
>
> <quote>
> Looks like each of the oninit processes implements a different (or
> parallel or redundant) part of the overall database server:
>
> -bash-3.2$ onstat -g sch
>
> IBM Informix Dynamic Server Version 11.50.UC8X6   -- On-Line -- Up 3 days
> 16:22:33 -- 230984 Kbytes
>
> VP Scheduler Statistics:
> vp    pid       class       semops    busy waits  spins/wait
> 1     24024     cpu         15202292  0           15205721
> 2     24066     adm         0         0           0
> 3     24067     LIC         18        0           18
> 4     24068     DBFNC       1         0           1
> 5     24069     lio         136050    0           0
> 6     24070     pio         1287      0           0
> 7     24071     aio         903428    0           0
> ...
>
> http://publib.boulder.ibm.com/infocenter/idshelp/v117/index.jsp
> </quote>
>
> Generally, yes, you're looking at something in the database generating
> lots of churn. THis could be CDRs, it could be CAR, it could also be AXL or
> high rate of change notifications.
>
> Looks like you would benefit from something like this:
> CLI show tech activesql
>
> from:
> CSCsz67357    Need an Informix profiler built into CLI
>
> Regards,
> Wes
>
> On Apr 5, 2013, at 11:20 AM, Erick Wellnitz <ewellnitzvoip at gmail.com>
> wrote:
>
> caroninit seems to be the biggest offender (by about 50x) in both disk
> writes and cpu usage.  Am I correct in assuming this has something to do
> with call detail records?
>
>
> On Fri, Apr 5, 2013 at 9:02 AM, Tom Piscitell (tpiscite) <
> tpiscite at cisco.com> wrote:
>
>> Erick,
>>
>> You can use the FIOR utility from the CLI to identify which processes are
>> writing to the disk.
>>
>> admin:utils fior
>>       utils fior disable
>>       utils fior enable
>>       utils fior list
>>       utils fior start
>>       utils fior status
>>       utils fior stop
>>       utils fior top
>>
>> Here is a typical use case:
>>
>> 1. Enable the FIOR utility before/during a time of High IO Wait
>>         admin:utils fior enable
>>         File I/O Statistics has been enabled.
>>         admin:utils fior start
>>         Loading fiostats module: ok
>>         Enabling fiostats : ok
>>         File I/O Statistics has been started.
>>
>> 2. Wait a couple minutes. FIOR will poll for data every 5 seconds I
>> believe. Then use utils fior top to see whats hitting the CPU the hardest:
>>
>> admin:utils fior top ?
>> Syntax:
>> utils fior top n sort_by [start=date-time] [stop=date-time]
>>
>>          n:            number of processes
>>          sort_by:      read, write, read-rate, write-rate
>>          date-time:    of the form %H:%M, %H:%M:%S
>>                                    %a,%H:%M, %a,%H:%M:%S
>>                                    %Y-%m-%d,%H:%M, %Y-%m-%d %H:%M:%S
>> Example:
>> admin:utils fior top 10 write start=2010-04-20 10:00:00 stop=2010-04-20
>> 10:30:00
>>
>> This of course won't tell you *why* a process is hitting the disk, but it
>> will at least show you who has the most read/writes. To answer the why
>> question you would need to look at traces for the offending process/service.
>>
>> HTH,
>> -Tom
>>
>> On Apr 4, 2013, at 5:43 PM, Erick Wellnitz <ewellnitzvoip at gmail.com>
>> wrote:
>>
>> > Hello all!
>> >
>> > I have a dual 4 core IBM 7835I3 which is my publisher.   One one core
>> of the first CPU the I/O Wait is through the roof.  RTMT shows that writes
>> to the hard drives are at between 600 and 700 MB/s which is exponentially
>> higher than the subscriber on the same model of hardware.
>> >
>> > Short of calling TAC is there any way to figure out what is causing the
>> extremely high volume of writes to the drives?  I already stopped most
>> traces and looking at the processes doesn't give any clues.
>> >
>> > Thanks again!
>> >
>> >
>> > _______________________________________________
>> > cisco-voip mailing list
>> > cisco-voip at puck.nether.net
>> > https://puck.nether.net/mailman/listinfo/cisco-voip
>>
>>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20130405/36f41558/attachment.html>