<div dir="ltr"><div>We have 10.5.x running with the same issue as well. frequently the tomcat hang on the publisher. </div><div><br>Finding is there's excessive AXL request from poorly programmed application which sort of DoS the server. </div><div>By right they should reuse AXL connection, instead the software keep on establishing around 5 connections per sec... </div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Dec 17, 2017 at 3:22 AM, Ryan Huff <span dir="ltr"><<a href="mailto:ryanhuff@outlook.com" target="_blank">ryanhuff@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
I cannot speak to Cisco’s implementation (if any Cisco CUCM devs are on here, feel free to chime in) however it’s been my experience as a programmer (long before I fell in love with UC) that tomcat, as a whole, is a rather leaky web server solution.
<div><br>
</div>
<div>When I last studied and researched this topic heavily, it seemed to be many community’s opinion this was largely due to buffer (memory) management (or lack of efficient management thereof).</div>
<div><br>
</div>
<div>Personally, I’d love to see Cisco decouple phone services from tomcat and package a more kick-ass web server with CUCM specifically for phone services, like Apache. I’ve suggested dual web server processes for a long time (for more reasons than just phone
services) but I’m just a voice in a sea of millions.</div>
<div><br>
</div>
<div>There is the old “trick” of tossing 2GB extra of RAM at each CUCM VM guest machine (if you can spare it). Generally speaking, it doesn’t solve the problem, but usually makes it less frequent.</div>
<div><br>
</div>
<div>At the very least, I would think in the modern mesh database versions of CUCM, phone services should be able to run on the subscribers. All that would likely take more re-architecting than I’m guessing Cisco is willing to invest in at this point.</div>
<div><br>
</div>
<div>To Charles’s point; predictive, scheduled tomcat restarts is the only way I’ve ever come to efficiently and effectively manage this when you have a cluster where tomcat is used heavily.</div><span class="HOEnZb"><font color="#888888">
<div><br>
</div>
<div>-Ryan</div></font></span><div><div class="h5">
<div>
<div><br>
On Dec 16, 2017, at 11:59 AM, Matt Jacobson <<a href="mailto:m4ttjacobson@gmail.com" target="_blank">m4ttjacobson@gmail.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr">
<div>
<div dir="auto">On the subject of tomcat performance and AXL requests, what is the recommended setup for TMS - CUCM integratio
<div class="gmail_default" style="font-family:trebuchet ms,sans-serif;display:inline">
n?? I have seen AXL requests from TMS to overwhelm tomcat causing admin interface to be unresponsive or crash until tomcat is restarted.</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">The TMS documentation is not specific about whether you add only one CUCM node or all nodes.
<div class="gmail_default" style="font-family:trebuchet ms,sans-serif;display:inline">
I plan to test this when I get a chance, but if you add all nodes, does TMS load balance these requests or just spam all the nodes in the same manner?</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">In the CUCM release notes it says this: </div>
<div dir="auto"><br>
</div>
AXL Requests to Unified CM Nodes<br>
<br>
If you run Cisco TelePresence Management Suite (TMS) for scheduling, then the node that you add it to sends multiple AXL queries to fetch endpoint information. Because of the load that TMS generates, we recommend that you do not configure other applications
that use AXL (such as Cisco Emergency Responder or Cisco Unified Attendant Console) to send AXL requests to these nodes.
<div dir="auto"><br>
</div>
<br>
<div class="gmail_quote">
<div>On Sat, Dec 16, 2017 at 10:50 Charles Goldsmith <<a href="mailto:wokka@justfamily.org" target="_blank">wokka@justfamily.org</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div>For us, we are restarting tomcat on the pub about once a week. Our administrative interface is used pretty heavily with MACD stuff. I've found that if we use one of the low utilization subs, we aren't having the issues.
<div><br>
</div>
<div>We can't restart tomcat that easily due to EM usage, and yes, we have a dedicated pub.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Dec 16, 2017 at 12:42 AM, Anthony Holloway <span>
<<a href="mailto:avholloway+cisco-voip@gmail.com" target="_blank">avholloway+cisco-voip@gmail.c<wbr>om</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div>Out of curiosity, how long had Tomcat been running before you restarted it?
<div><br>
</div>
<div>This isn't at you Terry, but in general.</div>
<div><br>
</div>
<div>Companies will spend a lot of money getting systems in place, but then completely forget that technology has a life cycle; leading towards a better experience. And no, I don't just mean upgrade to the latest shiny version. I mean, efficiency, features,
user experience, stability, scale, shorter MTTR.</div>
<div><br>
</div>
<div>Without being able to quantify it, I have seen more than a comfortable amount of environments
<b>without</b>: a pre-production environment, proper analytics, proper change control, a good monitoring solution (emails from RTMT don't count), resource usage monitoring, a good backup strategy, vmtools up to date, and anything other than just MACD work being
performed.</div>
<div><br>
</div>
<div>It's like there's this sole effort on "projects," and the old saying: "if isn't broke, don't fix it," wins again. We lose the chance to truly understand our systems, and therefore the chance to optimize them.</div>
<div><br>
</div>
<div>/rant</div>
<div><br>
</div>
<div><i>Disclaimer: Today was a long cutover, and I'm tired</i></div>
<div><br>
</div>
<div>PS Ryan amazes me too.
<div>
<div class="m_-4777699448332355825m_634227491076377914m_-5226261233555791437h5"><br>
<br>
<div class="gmail_quote">
<div>On Thu, Dec 14, 2017 at 10:32 PM Terry Oakley <<a href="mailto:Terry.Oakley@rdc.ab.ca" target="_blank">Terry.Oakley@rdc.ab.ca</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div>
<div id="m_-4777699448332355825m_634227491076377914m_-5226261233555791437m_2174437041092308227m_-9020250847252235192divtagdefaultwrapper" style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif;font-size:12pt">
<p style="margin-top:0px;margin-bottom:0px">Thank you again Ryan. I think I found the issue. One of the tests showed a problem with AXL services. Restarted Tomcat and we appear to be much better.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px"> <br>
</p>
</div>
<hr style="width:98%;display:inline-block">
<div id="m_-4777699448332355825m_634227491076377914m_-5226261233555791437m_2174437041092308227m_-9020250847252235192divRplyFwdMsg">
<font color="#000000" face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Terry Oakley<br>
<b>Sent:</b> Thursday, December 14, 2017 5:29:31 PM<br>
<b>To:</b> Ryan Huff</font></div>
</div>
<div>
<div id="m_-4777699448332355825m_634227491076377914m_-5226261233555791437m_2174437041092308227m_-9020250847252235192divRplyFwdMsg">
<font color="#000000" face="Calibri, sans-serif" style="font-size:11pt"><br>
<b>Cc:</b> <a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
<b>Subject:</b> Re: [cisco-voip] UC server performance and UCCX agent in reserve</font></div>
</div>
<div>
<div>
<div id="m_-4777699448332355825m_634227491076377914m_-5226261233555791437m_2174437041092308227m_-9020250847252235192x_divtagdefaultwrapper" style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif;font-size:12pt">
<p style="margin-top:0px;margin-bottom:0px">Thanks Ryan.. .I will have a look tonight..
<br>
</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">PS i don't know how you find all the time to respond to all of us but I am very thankful that you do. <span>😊</span><br>
</p>
</div>
<hr style="width:98%;display:inline-block">
<div id="m_-4777699448332355825m_634227491076377914m_-5226261233555791437m_2174437041092308227m_-9020250847252235192x_divRplyFwdMsg">
<font color="#000000" face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Ryan Huff <<a href="mailto:ryanhuff@outlook.com" target="_blank">ryanhuff@outlook.com</a>><br>
<b>Sent:</b> Thursday, December 14, 2017 5:26:53 PM<br>
<b>To:</b> Terry Oakley<br>
<b>Cc:</b> <a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
<b>Subject:</b> Re: [cisco-voip] UC server performance and UCCX agent in reserve</font>
<div> </div>
</div>
<div dir="auto">Just based on that description alone, I’d say it might be possible you have some LAN congestion?
<div>Everything you’re talking about here is riding http/https.</div>
<div>
<div><br>
</div>
<div>- Any recent QoS policy changes?</div>
<div><br>
</div>
<div>- Is other non-UC web traffic slower than normal from those PCs?</div>
<div><br>
</div>
<div>- Run <i>utils diagnose test</i> on the CLI of each server and see if you find any goodies ...</div>
<div><br>
<div>-Ryan</div>
<div><br>
On Dec 14, 2017, at 7:18 PM, Terry Oakley <<a href="mailto:Terry.Oakley@rdc.ab.ca" target="_blank">Terry.Oakley@rdc.ab.ca</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div id="m_-4777699448332355825m_634227491076377914m_-5226261233555791437m_2174437041092308227m_-9020250847252235192x_x_divtagdefaultwrapper" style="color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif;font-size:12pt">
<p style="margin-top:0px;margin-bottom:0px">For the past week and a bit I have noticed a decline in UC (Call Manager) response time when editing/adding a device. The message 'loading' stays on for 5 to 10 seconds or even longer. Page refresh is also really
slow. In looking at RTMT the CPU/Memory/disk space are all around 50% or less with no apparent spikes. Any suggestions on where this lag could be?</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">On another but may be related , a couple of our agents (but not all) both have had their phones restart while in use, and today both had their agent go into Reserved state for a couple of minutes before finally connecting
and allowing them service. Again any suggestions on where one would look would be appreciated.</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">UC 11.5 SU3</p>
<p style="margin-top:0px;margin-bottom:0px">UCCX 11.5</p>
<p style="margin-top:0px;margin-bottom:0px">IMP 11.5 SU3</p>
<p style="margin-top:0px;margin-bottom:0px">O365</p>
<p style="margin-top:0px;margin-bottom:0px">Unity Connection 11.5<br>
</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
<p style="margin-top:0px;margin-bottom:0px">Terry<br>
</p>
<p style="margin-top:0px;margin-bottom:0px"><br>
</p>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>______________________________<wbr>_________________</span><br>
<span>cisco-voip mailing list</span><br>
<span><a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a></span><br>
<span><a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank">https://puck.nether.net/mailma<wbr>n/listinfo/cisco-voip</a></span><br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
______________________________<wbr>_________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank" rel="noreferrer">https://puck.nether.net/mailma<wbr>n/listinfo/cisco-voip</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank" rel="noreferrer">https://puck.nether.net/mailma<wbr>n/listinfo/cisco-voip</a><br>
<br>
</blockquote>
</div>
<br>
</div>
______________________________<wbr>_________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank" rel="noreferrer">https://puck.nether.net/mailma<wbr>n/listinfo/cisco-voip</a><br>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>______________________________<wbr>_________________</span><br>
<span>cisco-voip mailing list</span><br>
<span><a href="mailto:cisco-voip@puck.nether.net" target="_blank">cisco-voip@puck.nether.net</a></span><br>
<span><a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank">https://puck.nether.net/<wbr>mailman/listinfo/cisco-voip</a></span><br>
</div>
</blockquote>
</div>
</div></div></div>
<br>______________________________<wbr>_________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank" rel="noreferrer">https://puck.nether.net/<wbr>mailman/listinfo/cisco-voip</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Regards,</div><div>Ki Wi</div></div></div>
</div>