[cisco-voip] Delay issues with CUCM862a Can someone point me in the right direction ??

Fri Apr 13 07:28:08 EDT 2012

Other complaints started coming in that worried me.
Besides the delayed audio the directory services menu options 4 and 5 intermittantly showed an error then it went away and showed up.
Directory searches intermittently take 8 seconds to appear.

I started monitoring cpu and I am seeing it spike to 96% and stay near 80-90% then dropping down to 70% then I saw a LowMemoryAlarm message...

I stopped troubleshooting the audio delay maybe this is a function of the cpu spike.

I did use the recommended ova file from the wiki site when I built the vm in v8.5 but when I upgraded to 8.6.2a I didn't add the extra memory.

I am going to recommend to my client to add more memory to the vm.

Greg

From: Wes Sisk [mailto:wsisk at cisco.com]
Sent: Thursday, April 12, 2012 11:42 AM
To: Gregory Wenzel
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Delay issues with CUCM862a Can someone point me in the right direction ??

StartMediaTransmission can only be sent when the peer device responds with the UDP/RTP IP and port where it wants to receive audio.  Investigate the signaling with the remote device. Either the remote device is slow to respond, the response is delayed back to ccm, or ccm is slow to propagate that information back to this phone device.

Regards,
Wes

On Apr 12, 2012, at 11:25 AM, Gregory Wenzel wrote:

Im using a tool called CCM Translator X that Paul Geralt wrote a few years back.

I am seeing a delay in signaling for the network that is in bldg 1.

I see the call state for normal openreceivechannel then its stationopenreceivechannelAckID and openreceivechannelAck are within a few milliseconds of each other

The normal time shows
OpenReceiveChannel at time 10.26.28.362
Then
startMediaTransmission at time 10.26.28.364
Then
Both stationopenreceivechannelAckID and openreceivechannelAck at time 10.26.28.367

I see a delay in the startMediaTransmision for this particular user has a gap between the openreceivechannel and the startMediaTransmission and its also missing the stationopenreceivechannelAckID and it seems there is a big gap in the startMedisTransmission

OpenReceiveChannel at time 10.26.48.523
Then
 Instead of startMediaTransmission starting I see that OpenReceiveChannelAck at 10.26.48.560
Then startMediaTransmisttion starts much later at 10.26.48.617

Greg

From: Wes Sisk [mailto:wsisk at cisco.com]
Sent: Thursday, April 12, 2012 10:30 AM
To: Gregory Wenzel
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] Delay issues with CUCM862a Can someone point me in the right direction ??

heuristics say:

upgrade is at least OS if not server hardware replacement.  did speed/duplex get set correctly? SCCP uses TCP which will retransmit lost packets but lost packets will manifest to user as delays.

Otherwise, for delay answering a bit more clarification would help:
* how are they answering calls? handset/headset/speaker/answer softkey/3rd party headset lifter/bluetooth headset/...
* does the phone promptly enter the offhook state? look at the lcd, does it look connected?
* does the delay happen in audio cut through? i.e. phone is clearly offhook but audio does not cut through for 5-10 sec?
* what model phones and what phone load?

ccm sdi and sdl traces can be used to investigate some of this. find offhook or answer from phone, then the callstate=5 connected,  and then the subsequent openreceivechannel/startmediatransmission sent to phone. if these are "timely" in the ccm sdi traces then the culprit is either:
1. delays in the network delivering these messages
2. delay in the far end device beginning to transmit or receive audio
=>for this check the ccm sdi traces for signaling with the far end device. any sign of delays in signaling?

after clearing everything possible in the SDI/SDL traces the next step would be a packet capture at the endpoint device to investigate signaling, tcp retransmits, and RTP stream statistics there.

For the phones there are a few known issues. From the TAC Hot Issues feed for Endpoints on Cisco.com<http://Cisco.com>:

Delay in starting RTP from 6941, Fixed CSCtc96381
RTP sequence number gaps result in one way audio on 7960, Fixed CSCtn69362
Minimize RTP Delay from HW side when answering call on RT-Lite, Open CSCtd57345
delay in audio cut through on 894x, Open CSCtu08287
6945 sccp: SW polling time improvement for RTL Clipping issue, Fixed CSCtq80427
Intermittent one way audio on 6901 phones, Fixed CSCts39379

( there is one for rtp seq number looping that caused lots of problems that i cannot find at the moment, i'll keep looking but this should get you started)

/wes

On Apr 11, 2012, at 6:24 PM, Gregory Wenzel wrote:

I've just started troubleshooting these issues for a client whom I upgraded from ccm4.x to cucm8.6.2a.
They claim the problem did not exist when ccm4.x was in production.
Can someone tell me which traces are best to tblshoot kind of issue, kick me in the right direction before I start RTMT on cucm and debug on the gateways.
The problem is not system wide and only occur in this one particular building.

Complaints
Delay in answering calls in Bldg 1
Delay in transferring call in Bldg 1
Problems started immediately after the upgrade but they are telling me now 4 months later
IP Comm - Web interface is slow Sometimes when making a call you do not hear the ring tone

Variables
Switches used  "EXTREME" brand non-Cisco and I do not have access to these switches either
No network changes occurred since the upgrade that the customer wants to admit to.
Has its own subscriber in bldg 1
Every other phone in the 2K phone sized network all register to the same subscriber do not have this issue
It uses a 2811 gateway running 12.4.24T that is on the other end of a 1 gig fiber link across the street and I have no access to the devices that connect the two building together.

What major differences between signaling with version ccm4 and cucm862a would cause this?
I'm looking through the bug finder now. I would love to tell the client it's their Extreme switches but why then would it work on the old windows version.

Maybe it didn't work and the problem just go more pronounced after the upgrade.

I'm just looking for someone to point me in the right direction...:)

TIA

Greg Wenzel

This message w/attachments (message) is solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. Unless specifically indicated, this message is not an offer to sell or a solicitation of any products.      _______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

This message w/attachments (message) is solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. 
Unless specifically indicated, this message is not an offer to sell or a solicitation of any products.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20120413/945527b6/attachment.html>