<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">What version are you? 7 or 8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.5pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Neal Haas<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> cisco-voip-bounces@puck.nether.net [mailto:cisco-voip-bounces@puck.nether.net]
<b>On Behalf Of </b>Wes Sisk<br>
<b>Sent:</b> Friday, April 05, 2013 9:09 AM<br>
<b>To:</b> Erick Wellnitz<br>
<b>Cc:</b> cisco-voip<br>
<b>Subject:</b> Re: [cisco-voip] high I/O Wait on one core<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">cmoninit is one of the informix database processes. There are are several processes that each do different roles. it's not simple to identify the exact role of a specific oninit process.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><quote><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Looks like each of the oninit processes implements a different (or parallel or redundant) part of the overall database server:<br>
<br>
-bash-3.2$ onstat -g sch<br>
<br>
IBM Informix Dynamic Server Version 11.50.UC8X6 -- On-Line -- Up 3 days 16:22:33 -- 230984 Kbytes<br>
<br>
VP Scheduler Statistics:<br>
vp pid class semops busy waits spins/wait<br>
1 24024 cpu 15202292 0 15205721 <br>
2 24066 adm 0 0 0 <br>
3 24067 LIC 18 0 18 <br>
4 24068 DBFNC 1 0 1 <br>
5 24069 lio 136050 0 0 <br>
6 24070 pio 1287 0 0 <br>
7 24071 aio 903428 0 0 <br>
...<br>
<br>
<a href="http://publib.boulder.ibm.com/infocenter/idshelp/v117/index.jsp">http://publib.boulder.ibm.com/infocenter/idshelp/v117/index.jsp</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"></quote><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Generally, yes, you're looking at something in the database generating lots of churn. THis could be CDRs, it could be CAR, it could also be AXL or high rate of change notifications.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Looks like you would benefit from something like this:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">CLI show tech activesql<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">from:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">CSCsz67357 Need an Informix profiler built into CLI <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Wes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Apr 5, 2013, at 11:20 AM, Erick Wellnitz <<a href="mailto:ewellnitzvoip@gmail.com">ewellnitzvoip@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">caroninit seems to be the biggest offender (by about 50x) in both disk writes and cpu usage. Am I correct in assuming this has something to do with call detail records?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Apr 5, 2013 at 9:02 AM, Tom Piscitell (tpiscite) <<a href="mailto:tpiscite@cisco.com" target="_blank">tpiscite@cisco.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Erick,<br>
<br>
You can use the FIOR utility from the CLI to identify which processes are writing to the disk.<br>
<br>
admin:utils fior<br>
utils fior disable<br>
utils fior enable<br>
utils fior list<br>
utils fior start<br>
utils fior status<br>
utils fior stop<br>
utils fior top<br>
<br>
Here is a typical use case:<br>
<br>
1. Enable the FIOR utility before/during a time of High IO Wait<br>
admin:utils fior enable<br>
File I/O Statistics has been enabled.<br>
admin:utils fior start<br>
Loading fiostats module: ok<br>
Enabling fiostats : ok<br>
File I/O Statistics has been started.<br>
<br>
2. Wait a couple minutes. FIOR will poll for data every 5 seconds I believe. Then use utils fior top to see whats hitting the CPU the hardest:<br>
<br>
admin:utils fior top ?<br>
Syntax:<br>
utils fior top n sort_by [start=date-time] [stop=date-time]<br>
<br>
n: number of processes<br>
sort_by: read, write, read-rate, write-rate<br>
date-time: of the form %H:%M, %H:%M:%S<br>
%a,%H:%M, %a,%H:%M:%S<br>
%Y-%m-%d,%H:%M, %Y-%m-%d %H:%M:%S<br>
Example:<br>
admin:utils fior top 10 write start=2010-04-20 10:00:00 stop=2010-04-20 10:30:00<br>
<br>
This of course won't tell you *why* a process is hitting the disk, but it will at least show you who has the most read/writes. To answer the why question you would need to look at traces for the offending process/service.<br>
<br>
HTH,<br>
-Tom<br>
<br>
On Apr 4, 2013, at 5:43 PM, Erick Wellnitz <<a href="mailto:ewellnitzvoip@gmail.com">ewellnitzvoip@gmail.com</a>> wrote:<br>
<br>
> Hello all!<br>
><br>
> I have a dual 4 core IBM 7835I3 which is my publisher. One one core of the first CPU the I/O Wait is through the roof. RTMT shows that writes to the hard drives are at between 600 and 700 MB/s which is exponentially higher than the subscriber on the same
model of hardware.<br>
><br>
> Short of calling TAC is there any way to figure out what is causing the extremely high volume of writes to the drives? I already stopped most traces and looking at the processes doesn't give any clues.<br>
><br>
> Thanks again!<br>
><br>
><br>
> _______________________________________________<br>
> cisco-voip mailing list<br>
> <a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-voip</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip">https://puck.nether.net/mailman/listinfo/cisco-voip</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>