[cisco-voip] Fidelus Attendant Console to Call Manager Issue

[Daniel Pagan]

Wrapping up on this thread.

The aCO application is designed to run numerous SQL queries to the AXL API in order to retrieve device and end user information from CUCM. One of these many queries requests information from the device table and performs a join to correlate end users to associated phone devices, and it was during this request that aCO began to show signs of “locking up”. This query is sent to AXL multiple times to ensure that all pertinent device and end user information is gathered.

The issue in this scenario is that when executing this query, which was broken down to multiple requests, CUCM ended up responding to aCO with a total of 70,000+ rows, delivered over 14 separate responses each containing 5,000 rows. TAC’s initial findings provided to Steven were incorrect – the AXL API did indeed receive the initial requests from aCO in a timely manner.

Reviewing AXL traces showed multiple rows were provided for the same device – the end user PKID changed between these rows, but multiple PKIDs were shared across many different devices. Digging in a bit deeper resulted in finding end users with device associations to 20,000+ IP phone devices – this drastically increased the number of rows returned for this device-to-enduser query. The impact is that an additional row is returned for every associated device-assigned DN. Once the request was completed and the console had the 70,000+ rows, the aCO console then took 10 minutes to process all the information locally, and at this point that the remaining SQL queries and JTAPI requests were very sluggish, contributing to the overall 20 minute loading time.

Collecting 70,000+ rows returned for the query mentioned above –  5-6 minutes
Processing the information provided via AXL – 10 minutes
Completing the remaining AXL queries, obtain JTAPI terminal, etc. – 5-7 minutes

Resolution is to either remove the 20,000+ device associations from the end user or delete the end user entirely and recreate it as an application user, then associating devices with the application user. This AXL query doesn’t poll for application users’ PKID only end users, but I cannot suggest associating thousands of devices with a single user, application or end.

The Akkadian support team has integrated these findings into their internal documentation. Hope this helps anyone who might experience this problem in the future.

- Daniel

From: cisco-voip [mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Daniel Pagan
Sent: Wednesday, February 26, 2014 11:35 AM
To: Casper, Steven; 'Erick'
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Fidelus Attendant Console to Call Manager Issue

I’ll see what I can contribute here as sort of an “outsider” as Fidelus is not part of Akkadian software support. Being sister companies, communicating with the Akkadian support group and developers is easily achieved, and as a member of the Fidelus UC managed services team I certainly hope to help in any way I can.

From what I can tell so far the CUCM node used for AXL and JTAPI/CTI communication wasn’t receiving requests from the workstation until ~15 minutes after the application was opened on the workstation, but if you have AXL traces (info) and CTIManager traces (detailed) I’ll review them offline, provide you with my findings, and make some suggestions.

First thing that comes to mind is taking the application out of the equation. Using a SOAP client like SoapUI, it’s possible to issue the same AXL requests to CUCM being made by aCO and seeing if you experience the same type of delay outside the console application.

I’ve emailed you offline to discuss the Akkadian case, trace files to collect, etc.

Thanks Steven!

- Daniel

From: cisco-voip [mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Casper, Steven
Sent: Wednesday, February 26, 2014 8:17 AM
To: 'Erick'
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Fidelus Attendant Console to Call Manager Issue

We can recreate the problem on a different PC on a different network, will try the ccm user login with the fidelus ID.

Steve
From: Erick [mailto:erickbee at gmail.com]
Sent: Tuesday, February 25, 2014 9:33 PM
To: Casper, Steven
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] Fidelus Attendant Console to Call Manager Issue

I haven't had this issue.

There is a 3.2.0 version of console that fixed issue with line statuses for us fidelus provided.

Try a different PC?

If you log into ccm user with fidelus user does it take long time to login? Might have to give user end user role briefly.

Sent from my iPhone

On Feb 25, 2014, at 7:35 PM, "Casper, Steven" <SCASPER at mtb.com<mailto:SCASPER at mtb.com>> wrote:
Cisco - System version: 8.6.2.22900-9

Akkadian console -  3.1.3.25025

From: Erick [mailto:erickbee at gmail.com]
Sent: Tuesday, February 25, 2014 8:17 PM
To: Casper, Steven
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] Fidelus Attendant Console to Call Manager Issue

What's the full version of call manager 8.6.2 and full version of fidelus?



Sent from my iPhone

On Feb 25, 2014, at 6:26 PM, "Casper, Steven" <SCASPER at mtb.com<mailto:SCASPER at mtb.com>> wrote:
Having an issue with a recently installed Fidelus attendant console where it takes 20 minutes to authenticate to the Call Manager (8.6.2). We have worked with TAC and Fidelus customer support on this all day to no avail. Network traces and Call Manager traces have been provided but Cisco and Fidelus have not found the issue.

What is really strange is we have a Cert cluster running the same level of software and the console logs in with only a 2 minute delay which is supposed to be normal. Of course the user database is a lot smaller but still 20 minutes is forever!

Has anyone else experienced this or have any ideas where the issue may be?

Thanks
Steve
************************************
This email may contain privileged and/or confidential information that is intended solely for the use of the addressee.  If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission.  If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy.  This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act.  You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information.
There are risks associated with the use of electronic transmission.  The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission.
************************************
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip
************************************
This email may contain privileged and/or confidential information that is intended solely for the use of the addressee.  If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission.  If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy.  This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act.  You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information.
There are risks associated with the use of electronic transmission.  The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission.
************************************
************************************
This email may contain privileged and/or confidential information that is intended solely for the use of the addressee.  If you are not the intended recipient or entity, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the transmission.  If you received this communication in error, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy.  This communication may contain nonpublic personal information about consumers subject to the restrictions of the Gramm-Leach-Bliley Act and the Sarbanes-Oxley Act.  You may not directly or indirectly reuse or disclose such information for any purpose other than to provide the services for which you are receiving the information.
There are risks associated with the use of electronic transmission.  The sender of this information does not control the method of transmittal or service providers and assumes no duty or obligation for the security, receipt, or third party interception of this transmission.
************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20140228/0384c555/attachment.html>