[cisco-voip] UC server performance and UCCX agent in reserve

Mon Dec 18 04:43:24 EST 2017

We have 10.5.x running with the same issue as well. frequently the tomcat
hang on the publisher.

Finding is there's excessive AXL request from poorly programmed application
which sort of DoS the server.
By right they should reuse AXL connection, instead the software keep on
establishing around 5 connections per sec...

On Sun, Dec 17, 2017 at 3:22 AM, Ryan Huff <ryanhuff at outlook.com> wrote:

> I cannot speak to Cisco’s implementation (if any Cisco CUCM devs are on
> here, feel free to chime in) however it’s been my experience as a
> programmer (long before I fell in love with UC) that tomcat, as a whole, is
> a rather leaky web server solution.
>
> When I last studied and researched this topic heavily, it seemed to be
> many community’s opinion this was largely due to buffer (memory) management
> (or lack of efficient management thereof).
>
> Personally, I’d love to see Cisco decouple phone services from tomcat and
> package a more kick-ass web server with CUCM specifically for phone
> services, like Apache. I’ve suggested dual web server processes for a long
> time (for more reasons than just phone services) but I’m just a voice in a
> sea of millions.
>
> There is the old “trick” of tossing 2GB extra of RAM at each CUCM VM guest
> machine (if you can spare it). Generally speaking, it doesn’t solve the
> problem, but usually makes it less frequent.
>
> At the very least, I would think in the modern mesh database versions of
> CUCM, phone services should be able to run on the subscribers. All that
> would likely take more re-architecting than I’m guessing Cisco is willing
> to invest in at this point.
>
> To Charles’s point; predictive, scheduled tomcat restarts is the only way
> I’ve ever come to efficiently and effectively manage this when you have a
> cluster where tomcat is used heavily.
>
> -Ryan
>
> On Dec 16, 2017, at 11:59 AM, Matt Jacobson <m4ttjacobson at gmail.com>
> wrote:
>
> On the subject of tomcat performance and AXL requests, what is the
> recommended setup for TMS - CUCM integratio
> n?? I have seen AXL requests from TMS to overwhelm tomcat causing admin
> interface to be unresponsive or crash until tomcat is restarted.
>
> The TMS documentation is not specific about whether you add only one CUCM
> node or all nodes.
> I plan to test this when I get a chance, but if you add all nodes, does
> TMS load balance these requests or just spam all the nodes in the same
> manner?
>
> In the CUCM release notes it says this:
>
> AXL Requests to Unified CM Nodes
>
> If you run Cisco TelePresence Management Suite (TMS) for scheduling, then
> the node that you add it to sends multiple AXL queries to fetch endpoint
> information. Because of the load that TMS generates, we recommend that you
> do not configure other applications that use AXL (such as Cisco Emergency
> Responder or Cisco Unified Attendant Console) to send AXL requests to these
> nodes.
>
>
> On Sat, Dec 16, 2017 at 10:50 Charles Goldsmith <wokka at justfamily.org>
> wrote:
>
>> For us, we are restarting tomcat on the pub about once a week.  Our
>> administrative interface is used pretty heavily with MACD stuff.  I've
>> found that if we use one of the low utilization subs, we aren't having the
>> issues.
>>
>> We can't restart tomcat that easily due to EM usage, and yes, we have a
>> dedicated pub.
>>
>> On Sat, Dec 16, 2017 at 12:42 AM, Anthony Holloway <
>> avholloway+cisco-voip at gmail.com> wrote:
>>
>>> Out of curiosity, how long had Tomcat been running before you restarted
>>> it?
>>>
>>> This isn't at you Terry, but in general.
>>>
>>> Companies will spend a lot of money getting systems in place, but then
>>> completely forget that technology has a life cycle; leading towards a
>>> better experience.  And no, I don't just mean upgrade to the latest shiny
>>> version.  I mean, efficiency, features, user experience, stability, scale,
>>> shorter MTTR.
>>>
>>> Without being able to quantify it, I have seen more than a comfortable
>>> amount of environments *without*: a pre-production environment, proper
>>> analytics, proper change control, a good monitoring solution (emails from
>>> RTMT don't count), resource usage monitoring, a good backup strategy,
>>> vmtools up to date, and anything other than just MACD work being performed.
>>>
>>> It's like there's this sole effort on "projects," and the old saying:
>>> "if isn't broke, don't fix it," wins again. We lose the chance to truly
>>> understand our systems, and therefore the chance to optimize them.
>>>
>>> /rant
>>>
>>> *Disclaimer: Today was a long cutover, and I'm tired*
>>>
>>> PS Ryan amazes me too.
>>>
>>>
>>> On Thu, Dec 14, 2017 at 10:32 PM Terry Oakley <Terry.Oakley at rdc.ab.ca>
>>> wrote:
>>>
>>>> Thank you again Ryan.   I think I found the issue.   One of the tests
>>>> showed a problem with AXL services.  Restarted Tomcat and we appear to be
>>>> much better.
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Terry Oakley
>>>> *Sent:* Thursday, December 14, 2017 5:29:31 PM
>>>> *To:* Ryan Huff
>>>>
>>>> *Cc:* cisco-voip at puck.nether.net
>>>> *Subject:* Re: [cisco-voip] UC server performance and UCCX agent in
>>>> reserve
>>>>
>>>> Thanks Ryan.. .I will have a look tonight..
>>>>
>>>>
>>>> PS i don't know how you find all the time to respond to all of us but I
>>>> am very thankful that you do.  😊
>>>> ------------------------------
>>>> *From:* Ryan Huff <ryanhuff at outlook.com>
>>>> *Sent:* Thursday, December 14, 2017 5:26:53 PM
>>>> *To:* Terry Oakley
>>>> *Cc:* cisco-voip at puck.nether.net
>>>> *Subject:* Re: [cisco-voip] UC server performance and UCCX agent in
>>>> reserve
>>>>
>>>> Just based on that description alone, I’d say it might be possible you
>>>> have some LAN congestion?
>>>> Everything you’re talking about here is riding http/https.
>>>>
>>>> - Any recent QoS policy changes?
>>>>
>>>> - Is other non-UC web traffic slower than normal from those PCs?
>>>>
>>>> - Run *utils diagnose test* on the CLI of each server and see if you
>>>> find any goodies ...
>>>>
>>>> -Ryan
>>>>
>>>> On Dec 14, 2017, at 7:18 PM, Terry Oakley <Terry.Oakley at rdc.ab.ca>
>>>> wrote:
>>>>
>>>> For the past week and a bit I have noticed a decline in UC (Call
>>>> Manager) response time when editing/adding a device.   The message
>>>> 'loading' stays on for 5 to 10 seconds or even longer.   Page refresh is
>>>> also really slow.   In looking at RTMT the CPU/Memory/disk space are all
>>>> around 50% or less with no apparent spikes.   Any suggestions on where this
>>>> lag could be?
>>>>
>>>>
>>>> On another but may be related , a couple of our agents (but not all)
>>>> both have had their phones restart while in use, and today both had their
>>>> agent go into Reserved state for a couple of minutes before finally
>>>> connecting and allowing them service.     Again any suggestions on where
>>>> one would look would be appreciated.
>>>>
>>>>
>>>> UC 11.5 SU3
>>>>
>>>> UCCX 11.5
>>>>
>>>> IMP 11.5 SU3
>>>>
>>>> O365
>>>>
>>>> Unity Connection 11.5
>>>>
>>>>
>>>> Terry
>>>>
>>>>
>>>> _______________________________________________
>>>> cisco-voip mailing list
>>>> cisco-voip at puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>
>>>> _______________________________________________
>>>> cisco-voip mailing list
>>>> cisco-voip at puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>
>>>
>>> _______________________________________________
>>> cisco-voip mailing list
>>> cisco-voip at puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>
>>>
>> _______________________________________________
>> cisco-voip mailing list
>> cisco-voip at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>

-- 
Regards,
Ki Wi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20171218/b37b266e/attachment.html>