[cisco-voip] UC server performance and UCCX agent in reserve

Sat Dec 16 21:22:54 EST 2017

I cannot speak to Cisco’s implementation (if any Cisco CUCM devs are on here, feel free to chime in) however it’s been my experience as a programmer (long before I fell in love with UC) that tomcat, as a whole, is a rather leaky web server solution.

When I last studied and researched this topic heavily, it seemed to be many community’s opinion this was largely due to buffer (memory) management (or lack of efficient management thereof).

Personally, I’d love to see Cisco decouple phone services from tomcat and package a more kick-ass web server with CUCM specifically for phone services, like Apache. I’ve suggested dual web server processes for a long time (for more reasons than just phone services) but I’m just a voice in a sea of millions.

There is the old “trick” of tossing 2GB extra of RAM at each CUCM VM guest machine (if you can spare it). Generally speaking, it doesn’t solve the problem, but usually makes it less frequent.

At the very least, I would think in the modern mesh database versions of CUCM, phone services should be able to run on the subscribers. All that would likely take more re-architecting than I’m guessing Cisco is willing to invest in at this point.

To Charles’s point; predictive, scheduled tomcat restarts is the only way I’ve ever come to efficiently and effectively manage this when you have a cluster where tomcat is used heavily.

-Ryan

On Dec 16, 2017, at 11:59 AM, Matt Jacobson <m4ttjacobson at gmail.com<mailto:m4ttjacobson at gmail.com>> wrote:

On the subject of tomcat performance and AXL requests, what is the recommended setup for TMS - CUCM integratio
n?? I have seen AXL requests from TMS to overwhelm tomcat causing admin interface to be unresponsive or crash until tomcat is restarted.

The TMS documentation is not specific about whether you add only one CUCM node or all nodes.
I plan to test this when I get a chance, but if you add all nodes, does TMS load balance these requests or just spam all the nodes in the same manner?

In the CUCM release notes it says this:

AXL Requests to Unified CM Nodes

If you run Cisco TelePresence Management Suite (TMS) for scheduling, then the node that you add it to sends multiple AXL queries to fetch endpoint information. Because of the load that TMS generates, we recommend that you do not configure other applications that use AXL (such as Cisco Emergency Responder or Cisco Unified Attendant Console) to send AXL requests to these nodes.

On Sat, Dec 16, 2017 at 10:50 Charles Goldsmith <wokka at justfamily.org<mailto:wokka at justfamily.org>> wrote:
For us, we are restarting tomcat on the pub about once a week.  Our administrative interface is used pretty heavily with MACD stuff.  I've found that if we use one of the low utilization subs, we aren't having the issues.

We can't restart tomcat that easily due to EM usage, and yes, we have a dedicated pub.

On Sat, Dec 16, 2017 at 12:42 AM, Anthony Holloway <avholloway+cisco-voip at gmail.com<mailto:avholloway+cisco-voip at gmail.com>> wrote:
Out of curiosity, how long had Tomcat been running before you restarted it?

This isn't at you Terry, but in general.

Companies will spend a lot of money getting systems in place, but then completely forget that technology has a life cycle; leading towards a better experience.  And no, I don't just mean upgrade to the latest shiny version.  I mean, efficiency, features, user experience, stability, scale, shorter MTTR.

Without being able to quantify it, I have seen more than a comfortable amount of environments without: a pre-production environment, proper analytics, proper change control, a good monitoring solution (emails from RTMT don't count), resource usage monitoring, a good backup strategy, vmtools up to date, and anything other than just MACD work being performed.

It's like there's this sole effort on "projects," and the old saying: "if isn't broke, don't fix it," wins again. We lose the chance to truly understand our systems, and therefore the chance to optimize them.

/rant

Disclaimer: Today was a long cutover, and I'm tired

PS Ryan amazes me too.

On Thu, Dec 14, 2017 at 10:32 PM Terry Oakley <Terry.Oakley at rdc.ab.ca<mailto:Terry.Oakley at rdc.ab.ca>> wrote:

Thank you again Ryan.   I think I found the issue.   One of the tests showed a problem with AXL services.  Restarted Tomcat and we appear to be much better.

________________________________
From: Terry Oakley
Sent: Thursday, December 14, 2017 5:29:31 PM
To: Ryan Huff

Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] UC server performance and UCCX agent in reserve

Thanks Ryan.. .I will have a look tonight..

PS i don't know how you find all the time to respond to all of us but I am very thankful that you do.  😊

________________________________
From: Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>>
Sent: Thursday, December 14, 2017 5:26:53 PM
To: Terry Oakley
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] UC server performance and UCCX agent in reserve

Just based on that description alone, I’d say it might be possible you have some LAN congestion?
Everything you’re talking about here is riding http/https.

- Any recent QoS policy changes?

- Is other non-UC web traffic slower than normal from those PCs?

- Run utils diagnose test on the CLI of each server and see if you find any goodies ...

-Ryan

On Dec 14, 2017, at 7:18 PM, Terry Oakley <Terry.Oakley at rdc.ab.ca<mailto:Terry.Oakley at rdc.ab.ca>> wrote:

For the past week and a bit I have noticed a decline in UC (Call Manager) response time when editing/adding a device.   The message 'loading' stays on for 5 to 10 seconds or even longer.   Page refresh is also really slow.   In looking at RTMT the CPU/Memory/disk space are all around 50% or less with no apparent spikes.   Any suggestions on where this lag could be?

On another but may be related , a couple of our agents (but not all) both have had their phones restart while in use, and today both had their agent go into Reserved state for a couple of minutes before finally connecting and allowing them service.     Again any suggestions on where one would look would be appreciated.

UC 11.5 SU3

UCCX 11.5

IMP 11.5 SU3

O365

Unity Connection 11.5

Terry

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20171217/9af08330/attachment.html>