<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<base href="x-msg://71/"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Other complaints started coming in that worried me.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Besides the delayed audio the directory services menu options 4 and 5 intermittantly showed an error then it went away and showed up.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Directory searches intermittently take 8 seconds to appear.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I started monitoring cpu and I am seeing it spike to 96% and stay near 80-90% then dropping down to 70% then I saw a LowMemoryAlarm message…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I stopped troubleshooting the audio delay maybe this is a function of the cpu spike.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I did use the recommended ova file from the wiki site when I built the vm in v8.5 but when I upgraded to 8.6.2a I didn’t add the extra memory.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I am going to recommend to my client to add more memory to the vm.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Greg</span><span style="font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Wes Sisk [mailto:wsisk@cisco.com]
<br>
<b>Sent:</b> Thursday, April 12, 2012 11:42 AM<br>
<b>To:</b> Gregory Wenzel<br>
<b>Cc:</b> cisco-voip@puck.nether.net<br>
<b>Subject:</b> Re: [cisco-voip] Delay issues with CUCM862a Can someone point me in the right direction ??<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">StartMediaTransmission can only be sent when the peer device responds with the UDP/RTP IP and port where it wants to receive audio. Investigate the signaling with the remote device. Either the remote device is slow to respond, the response
is delayed back to ccm, or ccm is slow to propagate that information back to this phone device.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Wes<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Apr 12, 2012, at 11:25 AM, Gregory Wenzel wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<span class="apple-style-span"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif""><o:p></o:p></span></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Im using a tool called CCM Translator X that Paul Geralt wrote a few years back.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I am seeing a delay in signaling for the network that is in bldg 1.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I see the call state for normal openreceivechannel then its stationopenreceivechannelAckID and openreceivechannelAck are within a few milliseconds of each other</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The normal time shows</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">OpenReceiveChannel at time 10.26.28.362</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Then</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">startMediaTransmission at time 10.26.28.364</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Then</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Both stationopenreceivechannelAckID and openreceivechannelAck at time 10.26.28.367</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I see a delay in the startMediaTransmision for this particular user has a gap between the openreceivechannel and the startMediaTransmission and its also missing
the stationopenreceivechannelAckID and it seems there is a big gap in the startMedisTransmission</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">OpenReceiveChannel at time 10.26.48.523</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Then</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> Instead of startMediaTransmission starting I see that OpenReceiveChannelAck at 10.26.48.560</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Then startMediaTransmisttion starts much later at 10.26.48.617</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Calibri","sans-serif";color:#1F497D">Greg</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><o:p></o:p></p>
</div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in;border-width:initial;border-color:initial">
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span class="apple-converted-space"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Wes
Sisk [<a href="mailto:wsisk@cisco.com">mailto:wsisk@cisco.com</a>]<span class="apple-converted-space"> </span><br>
<b>Sent:</b><span class="apple-converted-space"> </span>Thursday, April 12, 2012 10:30 AM<br>
<b>To:</b><span class="apple-converted-space"> </span>Gregory Wenzel<br>
<b>Cc:</b><span class="apple-converted-space"> </span><a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
<b>Subject:</b><span class="apple-converted-space"> </span>Re: [cisco-voip] Delay issues with CUCM862a Can someone point me in the right direction ??</span><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">heuristics say:<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">upgrade is at least OS if not server hardware replacement. did speed/duplex get set correctly? SCCP uses TCP which will retransmit lost packets but lost packets will manifest to user as delays.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">Otherwise, for delay answering a bit more clarification would help:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">* how are they answering calls? handset/headset/speaker/answer softkey/3rd party headset lifter/bluetooth headset/...<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">* does the phone promptly enter the offhook state? look at the lcd, does it look connected?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">* does the delay happen in audio cut through? i.e. phone is clearly offhook but audio does not cut through for 5-10 sec?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">* what model phones and what phone load?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">ccm sdi and sdl traces can be used to investigate some of this. find offhook or answer from phone, then the callstate=5 connected, and then the subsequent openreceivechannel/startmediatransmission sent to phone. if these are "timely" in
the ccm sdi traces then the culprit is either:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">1. delays in the network delivering these messages<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">2. delay in the far end device beginning to transmit or receive audio<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">=>for this check the ccm sdi traces for signaling with the far end device. any sign of delays in signaling?<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">after clearing everything possible in the SDI/SDL traces the next step would be a packet capture at the endpoint device to investigate signaling, tcp retransmits, and RTP stream statistics there.<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">For the phones there are a few known issues. From the TAC Hot Issues feed for Endpoints on<span class="apple-converted-space"> </span><a href="http://Cisco.com">Cisco.com</a>:<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><br>
Delay in starting RTP from 6941, Fixed CSCtc96381<br>
RTP sequence number gaps result in one way audio on 7960, Fixed CSCtn69362<br>
Minimize RTP Delay from HW side when answering call on RT-Lite, Open CSCtd57345<br>
delay in audio cut through on 894x, Open CSCtu08287<br>
6945 sccp: SW polling time improvement for RTL Clipping issue, Fixed CSCtq80427<br>
Intermittent one way audio on 6901 phones, Fixed CSCts39379<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">( there is one for rtp seq number looping that caused lots of problems that i cannot find at the moment, i'll keep looking but this should get you started)<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">/wes<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Apr 11, 2012, at 6:24 PM, Gregory Wenzel wrote:<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">I've just started troubleshooting these issues for a client whom I upgraded from ccm4.x to cucm8.6.2a.<span class="apple-converted-space"> </span><br>
They claim the problem did not exist when ccm4.x was in production.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">Can someone tell me which traces are best to tblshoot kind of issue, kick me in the right direction before I start RTMT on cucm and debug on the gateways.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">The problem is not system wide and only occur in this one particular building.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><b><span style="font-family:"Arial","sans-serif"">Complaints</span></b><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">Delay in answering calls in Bldg 1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">Delay in transferring call in Bldg 1</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">Problems started immediately after the upgrade but they are telling me now 4 months later</span><o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">IP Comm - Web interface is slow Sometimes when making a call you do not hear the ring tone</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:5.0pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><b>Variables</b><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">Switches used "EXTREME" brand non-Cisco and I do not have access to these switches either</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">No network changes occurred since the upgrade that the customer wants to admit to.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">Has its own subscriber in bldg 1</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">Every other phone in the 2K phone sized network all register to the same subscriber do not have this issue</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">It uses a 2811 gateway running 12.4.24T that is on the other end of a 1 gig fiber link across the street and I have no access to the devices that connect the two
building together.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif""> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif""> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif""> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">What major differences between signaling with version ccm4 and cucm862a would cause this?</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">I’m looking through the bug finder now. I would love to tell the client it’s their Extreme switches but why then would it work on the old windows version.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif""> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">Maybe it didn’t work and the problem just go more pronounced after the upgrade.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif""> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif""> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"><span style="font-family:"Arial","sans-serif"">I’m just looking for someone to point me in the right direction...:)</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:5.0pt">TIA<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Greg Wenzel</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><br>
This message w/attachments (message) is solely for the use of the intended recipient(s) and may contain information that is privileged, confidential or proprietary. If you are not an intended recipient, please notify the sender, and then please delete and destroy
all copies and attachments, and be advised that any review or dissemination of, or the taking of any action in reliance on, the information contained in or attached to this message is prohibited. Unless specifically indicated, this message is not an offer
to sell or a solicitation of any products. _______________________________________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip">https://puck.nether.net/mailman/listinfo/cisco-voip</a><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>