[cisco-voip] CUCM ICT MasterSlaveDetermination Question
Anthony Holloway
avholloway+cisco-voip at gmail.com
Wed Sep 18 23:12:59 EDT 2013
Ryan,
Thank you for the reply.
Without reading too much into what you are saying about the "wait for
far-end TCS", did you mean to say that it's configurable and enabled by
default? I see on H.323 gateways you can configure a similar if not exact
setting called "wait for far end H245 capabilities", but on an ICT no such
option exists.
Also, when you say "let the side with the media resources send TCS last" in
the scenario I'm in, both sides have media resources, and both sides in
fact invoke media resources. PErhaps I overlooked a fundamental design
characteristic of H.323 ICT's where this is normal and expected behavior.
Hence we see Cisco's best practice switched to SIP trunks.
Thanks for the feedback Ryan, I appreciate it.
On Wed, Sep 18, 2013 at 3:38 PM, Ryan Ratliff (rratliff) <rratliff at cisco.com
> wrote:
> I believe this kind of situation is why we have the "wait for far-end
> TCS" option, which is enabled by default.
>
> Let the side with the media resources be the last one to send its TCS.
> That way when it receives a TCS that will have a caps mismatch the
> transcoder can be set up and only a matching caps set can be sent back the
> other way.
>
> -Ryan
>
> On Sep 18, 2013, at 1:18 PM, Anthony Holloway <
> avholloway+cisco-voip at gmail.com> wrote:
>
> Great resource you referenced there Jason. ipexpert certainly thinks its
> random. In which case, how do you design your media resources to
> effectively manage codecs? Just double up on each side? Thanks again for
> your feedback. Let's both wait to see what becomes of this discussion.
>
>
> On Wed, Sep 18, 2013 at 10:38 AM, Jason Aarons (AM) <
> jason.aarons at dimensiondata.com> wrote:
>
>> MSD selection algorithm both ICT and to a H323 gateway. I too would
>> love to know the algorithm. Seems to be more random.****
>>
>> I ran about 10 calls out a H323 IOS gateway and whom was Master seemed to
>> be random. Can we influence Master selection somehow?****
>>
>> ** **
>>
>>
>> http://blog.ipexpert.com/2010/08/04/gatekeeeperh323-signaling-ras-h225-h245/
>> ****
>>
>> See the H245 section.****
>>
>> ** **
>>
>> *From:* cisco-voip [mailto:cisco-voip-bounces at puck.nether.net] *On
>> Behalf Of *Anthony Holloway
>> *Sent:* Wednesday, September 18, 2013 11:03 AM
>> *To:* Cisco VoIP Group
>> *Subject:* [cisco-voip] CUCM ICT MasterSlaveDetermination Question****
>>
>> ** **
>>
>> ** **
>>
>> All,****
>>
>> I am troubleshooting a complex call flow with multiple clusters all
>> connected via Non-Gatekeeper Controlled H.323 ICT's, and have narrowed down
>> the failure to a region misconfiguration between a trunk and its transcoder.
>> ****
>>
>> My question to the group is about the H.245 MasterSlaveDetermination
>> process, and how CUCM handles this between two CUCM's. Because, as you
>> might see below, it seems to be broken, or at least unpredictable. Or
>> maybe my lack of understanding is blinding me to the obvious answer.****
>>
>> First, let me start with a graphic of the H.225/H.245 messaging between
>> two clusters. What you will see here is that TCS is sent in both
>> directions, and the the initiating side sends an H.245 endSession to the
>> receiving side, before the MasterSlaveDetermination is sent or received by
>> either party.
>>
>> ****
>>
>> <image001.png>****
>>
>> ** **
>>
>> What the graphic doesn't show, are the details within the messages, but
>> as you may know, the H.225 messaging is littered with customer specific
>> details, so it's a bit of a challenge to simply just post it all to this
>> mailing list. Below are some of the details to help you understand the
>> flow. Please let me know if you need more specific details.****
>>
>> We'll start with the TCS message sent by the .9 host to the .13 host****
>>
>> 05:37:28.999 |
>> H245ASN - TtPid=(5551) -Outgoing #80462 -value
>> MultimediaSystemControlMessage ::= request : terminalCapabilitySet :
>> {
>> sequenceNumber 1,
>> protocolIdentifier { 0 0 8 245 0 10 },
>> multiplexCapability h2250Capability :
>> {
>> maximumAudioDelayJitter 60,
>> receiveMultipointCapability
>> {
>> multicastCapability FALSE,
>> multiUniCastConference FALSE,
>> mediaDistributionCapability
>> {
>> {
>> centralizedControl FALSE,
>> distributedControl FALSE,
>> centralizedAudio FALSE,
>> distributedAudio FALSE,
>> centralizedVideo FALSE,
>> distributedVideo FALSE
>> }
>> }
>> },
>> transmitMultipointCapability
>> {
>> multicastCapability FALSE,
>> multiUniCastConference FALSE,
>> mediaDistributionCapability
>> {
>> {
>> centralizedControl FALSE,
>> distributedControl FALSE,
>> centralizedAudio FALSE,
>> distributedAudio FALSE,
>> centralizedVideo FALSE,
>> distributedVideo FALSE
>> }
>> }
>> },
>> receiveAndTransmitMultipointCapability
>> {
>> multicastCapability FALSE,
>> multiUniCastConference FALSE,
>> mediaDistributionCapability
>> {
>> {
>> centralizedControl FALSE,
>> distributedControl FALSE,
>> centralizedAudio FALSE,
>> distributedAudio FALSE,
>> centralizedVideo TRUE,
>> distributedVideo FALSE
>> }
>> }
>> },
>> mcCapability
>> {
>> centralizedConferenceMC FALSE,
>> decentralizedConferenceMC FALSE
>> },
>> rtcpVideoControlCapability FALSE,
>> mediaPacketizationCapability
>> {
>> h261aVideoPacketization FALSE
>> },
>> logicalChannelSwitchingCapability FALSE,
>> t120DynamicPortCapability FALSE
>> },
>> capabilityTable
>> {
>> {
>> capabilityTableEntryNumber 1,
>> capability nonStandard :
>> {
>> nonStandardIdentifier h221NonStandard :
>> {
>> t35CountryCode 181,
>> t35Extension 0,
>> manufacturerCode 18
>> },
>> data '010000001400000000000000C0CEA6B5'H
>> }
>> },
>> {
>> capabilityTableEntryNumber 2,
>> capability genericControlCapability :
>> {
>> capabilityIdentifier standard : { 0 0 8 323 1 3 1 }
>> }
>> },
>> {
>> capabilityTableEntryNumber 3,
>> capability receiveAudioCapability : g729wAnnexB : 6
>> },
>> {
>> capabilityTableEntryNumber 4,
>> capability receiveAudioCapability : g729AnnexAwAnnexB : 6
>> },
>> {
>> capabilityTableEntryNumber 5,
>> capability receiveAudioCapability : g729 : 6
>> },
>> {
>> capabilityTableEntryNumber 6,
>> capability receiveAudioCapability : g729AnnexA : 6
>> },
>> {
>> capabilityTableEntryNumber 7,
>> capability receiveAndTransmitUserInputCapability : dtmf : NULL
>> },
>> {
>> capabilityTableEntryNumber 8,
>> capability receiveAndTransmitUserInputCapability : basicString
>> : NULL
>> },
>> {
>> capabilityTableEntryNumber 44,
>> capability receiveAndTransmitUserInputCapability : hookflash :
>> NULL
>> }
>> },
>> capabilityDescriptors
>> {
>> {
>> capabilityDescriptorNumber 0,
>> |*^*^*****
>>
>> ** **
>>
>> This is the TCS received on the .9 host from the .13 host****
>>
>> ** **
>>
>> 05:37:29.211 |
>> H245ASN - TtPid=(5551) [0xb0efe090 3040 bytes] -Incoming #103057 -value
>> MultimediaSystemControlMessage ::= request : terminalCapabilitySet :
>> {
>> sequenceNumber 1,
>> protocolIdentifier { 0 0 8 245 0 10 },
>> multiplexCapability h2250Capability :
>> {
>> maximumAudioDelayJitter 60,
>> receiveMultipointCapability
>> {
>> multicastCapability FALSE,
>> multiUniCastConference FALSE,
>> mediaDistributionCapability
>> {
>> {
>> centralizedControl FALSE,
>> distributedControl FALSE,
>> centralizedAudio FALSE,
>> distributedAudio FALSE,
>> centralizedVideo FALSE,
>> distributedVideo FALSE
>> }
>> }
>> },
>> transmitMultipointCapability
>> {
>> multicastCapability FALSE,
>> multiUniCastConference FALSE,
>> mediaDistributionCapability
>> {
>> {
>> centralizedControl FALSE,
>> distributedControl FALSE,
>> centralizedAudio FALSE,
>> distributedAudio FALSE,
>> centralizedVideo FALSE,
>> distributedVideo FALSE
>> }
>> }
>> },
>> receiveAndTransmitMultipointCapability
>> {
>> multicastCapability FALSE,
>> multiUniCastConference FALSE,
>> mediaDistributionCapability
>> {
>> {
>> centralizedControl FALSE,
>> distributedControl FALSE,
>> centralizedAudio FALSE,
>> distributedAudio FALSE,
>> centralizedVideo TRUE,
>> distributedVideo FALSE
>> }
>> }
>> },
>> mcCapability
>> {
>> centralizedConferenceMC FALSE,
>> decentralizedConferenceMC FALSE
>> },
>> rtcpVideoControlCapability FALSE,
>> mediaPacketizationCapability
>> {
>> h261aVideoPacketization FALSE
>> },
>> logicalChannelSwitchingCapability FALSE,
>> t120DynamicPortCapability FALSE
>> },
>> capabilityTable
>> {
>> {
>> capabilityTableEntryNumber 1,
>> capability nonStandard :
>> {
>> nonStandardIdentifier h221NonStandard :
>> {
>> t35CountryCode 181,
>> t35Extension 0,
>> manufacturerCode 18
>> },
>> data '010000001400000000000000C0EE86B6'H
>> }
>> },
>> {
>> capabilityTableEntryNumber 2,
>> capability genericControlCapability :
>> {
>> capabilityIdentifier standard : { 0 0 8 323 1 3 1 }
>> }
>> },
>> {
>> capabilityTableEntryNumber 3,
>> capability receiveAudioCapability : g711Ulaw64k : 30
>> },
>> {
>> capabilityTableEntryNumber 4,
>> capability receiveAudioCapability : g711Alaw64k : 30
>> },
>> {
>> capabilityTableEntryNumber 5,
>> capability receiveAndTransmitUserInputCapability : dtmf : NULL
>> },
>> {
>> capabilityTableEntryNumber 6,
>> capability receiveAndTransmitUserInputCapability : basicString
>> : NULL
>> },
>> {
>> capabilityTableEntryNumber 44,
>> capability receiveAndTransmitUserInputCapability : hookflash :
>> NULL
>> }
>> },
>> capabilityDescriptors
>> {
>> {
>> capabilityDescriptorNumber 0,
>> simultaneousCapabilities
>> {
>> {
>> 3,
>> 4
>> },
>> {
>> 5,
>> 6
>> },
>> {
>> 44
>> }
>> }
>> }
>> |*^*^*****
>>
>> ** **
>>
>> What we can see now is a mismatch in supported capabilities: g729 versus
>> g711. Now, it is my understanding that during a dispute of capabilities,
>> the Master gets to dictate the capabilities to use.****
>>
>> The next two messages in the above ladder diagram are both sides ACKing
>> each other's TCS's.****
>>
>> 05:37:29.212 |
>> H245ASN - TtPid=(5551) -Outgoing #80463 -value
>> MultimediaSystemControlMessage ::= response : terminalCapabilitySetAck :
>> {
>> sequenceNumber 1
>> }
>> |*^*^*
>> 05:37:29.213 |
>> H245ASN - TtPid=(5551) [0xb0e99b58 1444 bytes] -Incoming #103058 -value
>> MultimediaSystemControlMessage ::= response : terminalCapabilitySetAck :
>> {
>> sequenceNumber 1
>> }
>> |*^*^*****
>>
>> ** **
>>
>> However, in the CUCM traces, between the Outgoing TCS ACK and the
>> Incoming TCS ACK, we see CUCM Media Resource Manager comparing codecs and
>> regions, and deciding it needs to invoke a transcoder. Even though
>> MasterSlaveDetermination has not yet happen.****
>>
>> 05:37:29.213 |DET-MediaManager-(22410)::preCheckCapabilities,
>> region1=ICT-REG, region2=ICT-REG, Pty1 capCount=4 (Cap,ptime)= (15,60)
>> (16,60) (11,60) (12,60), Pty2 capCount=2 (Cap,ptime)= (4,30)
>> (2,30)|4,100,21,5551.2^X.X.X.13^Port 57377
>> 05:37:29.213 |RegionsServer::MatchCapabilities -- kbps=8, capACount=4,
>> capBCount=2|*^*^*
>> 05:37:29.213 |DET-MediaManager-(22410)::preCheckCapabilities, caps
>> mismatch! Xcoder Reqd. kbps(8), filtered A[capCount=4 (Cap,ptime)= (15,60)
>> (16,60) (11,60) (12,60)], B[capCount=0 (Cap,ptime)=] allowMTP=0
>> numXcoderRequired=1 xcodingSide=2|4,100,21,5551.2^X.X.X.13^Port 57377
>> 05:37:29.213 |DET-MediaManager-(22410)::prepareInitialConnectionList,
>> Party1CapCount=4 Party2CapCount=2 XcoderRequired=1 xcodingSide=2
>> allowMTP=0|4,100,21,5551.2^X.X.X.13^Port 57377****
>>
>> ** **
>>
>> The timing in the logs, and unfortunately I do not have a PCAP to compare
>> against, is such that the transcoder fails to be invoked due to a region
>> misconfiguration between the ICT and the transcoder. The classic mixup:
>> "We need to transcode between g711 and g729, but the transcoder region to
>> the g711 device is set for 8kbps."****
>>
>> 05:37:29.213 |MediaResourceCdpc::getNextMTPDevice MtpDevice=XCODE1
>> Cepn=57d029f7-023b-2605-459d-af4afab55b6e devCap=0
>> resoureCount=1|4,100,21,5551.2^X.X.X.13^Port 57377
>> 05:37:29.213 |MediaResourceCdpc(21468)::sendMtpAllocateRequestToDevice
>> MtpResource=XCODE1
>> Cepn=57d029f7-023b-2605-459d-af4afab55b6e|4,100,21,5551.2^X.X.X.13^Port
>> 57377
>> 05:37:29.228
>> |MediaResourceCdpc(21468)::resource_rsvp_AllocateMtpResourceErr
>> Device=XCODE1 deviceCapIntersec=0|4,100,21,5551.2^X.X.X.13^Port 57377
>> 05:37:29.228 |MediaResourceCdpc(21468)::adjustDeviceTblXcoder -
>> mCepn=57d029f7-023b-2605-459d-af4afab55b6e, allocateErrBitset=0x44,
>> devCapIntersect=0x0, mandatoryCaps=0x0,
>> resourceCount=1|4,100,21,5551.2^X.X.X.13^Port 57377
>> 05:37:29.228 |MediaResourceCdpc(21468)::adjustDeviceTblXcoder - Codec
>> Mismatch|4,100,21,5551.2^X.X.X.13^Port 57377****
>>
>> ** **
>>
>> So it's at this point in the logs that we see CUCM fail the call due to
>> no media resources available.****
>>
>> 05:37:29.233 |!!ERROR!!
>> -MediaManager-(22410)::disconnOnResourceAllocationFailure, ERROR
>> disconnOnResourceAllocationFailure - fails to allocate
>> MTP/XCoder,connCount=2|4,100,21,5551.2^X.X.X.13^Port 57377****
>>
>> ** **
>>
>> It's now at .236 that we see the .9 host send the H.245 endSession to the
>> .13 host****
>>
>> 05:37:29.235 |
>> H245ASN - TtPid=(5550) -Outgoing #80464 -value
>> MultimediaSystemControlMessage ::= command : endSessionCommand : disconnect
>> : NULL
>> |*^*^*****
>>
>> ** **
>>
>> So that's why the call failed: .9 failed to invoke the media resource,
>> and thus sent an endSession.****
>>
>> ** **
>>
>> What we don't see in the diagram above, is that in the .13 logs, it too
>> was trying to invoke a transcoder based on the TCS it received: g729,
>> because the endpoint: UCCX, is hard coded to g711.****
>>
>> 22:37:29.114 |DET-MediaManager-(46351)::preCheckCapabilities,
>> region1=ICT-REG, region2=UCCX-REG, Pty1 capCount=4 (Cap,ptime)= (15,60)
>> (16,60) (11,60) (12,60), Pty2 capCount=2 (Cap,ptime)= (4,30)
>> (2,30)|2,100,21,7815.2^X.X.X.13^Port 57377
>> 22:37:29.114 |RegionsServer::MatchCapabilities -- kbps=64, capACount=4,
>> capBCount=2|*^*^*
>> 22:37:29.114 |DET-MediaManager-(46351)::preCheckCapabilities, caps
>> mismatch! Xcoder Reqd. kbps(64), filtered A[capCount=4 (Cap,ptime)= (15,60)
>> (16,60) (11,60) (12,60)], B[capCount=2 (Cap,ptime)= (4,30) (2,30)]
>> allowMTP=0 numXcoderRequired=1 ****
>>
>> ** **
>>
>> However, it invoked the transcoder successfully, and since the transcoder
>> is 64kbps to UCCX and 8kbps to the ICT, this is why we'll see another TCS
>> sent from the .13 host with g729 as its capabilities.****
>>
>> 22:37:29.115 |MediaResourceCdpc::getNextMTPDevice MtpDevice=XCODE2
>> Cepn=26c772aa-3d19-4289-7dce-1d48d19807d7 devCap=0
>> resoureCount=1|2,100,21,7815.2^X.X.X.13^Port 57377
>> 22:37:29.115 |MediaResourceCdpc(38981)::sendMtpAllocateRequestToDevice
>> MtpResource=XCODE2
>> Cepn=26c772aa-3d19-4289-7dce-1d48d19807d7|2,100,21,7815.2^X.X.X.13^Port
>> 57377
>> 22:37:29.115
>> |MediaTerminationPointControl(1)::waiting_AllocateMtpResourceReq -
>> (capCount,region),A(4,ICT-REG),B(2,UCCX-REG), reqDevCap=0x0,
>> reqMandatoryCaps=0x0, supDevCap=0x129, passthru=0,
>> resourceCount=1|2,100,21,7815.2^X.X.X.13^Port 57377
>> 22:37:29.115 |MediaTerminationPointControl(1)::getResourcesAllocated --
>> DeviceName=XCODE2 Ci=42721675 ResourceCount=1|2,100,21,7815.2^X.X.X.13^Port
>> 57377
>> 22:37:29.115 |MediaTerminationPointControl(1)::getResourcesAllocated --
>> Logging RegionA=ICT-REG Caps and MTP/XCoder Region=XIR-REG
>> Caps|2,100,21,7815.2^X.X.X.13^Port 57377
>> 22:37:29.115 |MediaTerminationPointControl(1)::logCapabilitiesinTrace --
>> MTP/XCoder Device Caps = 6 4 2 16 11 12 257 259 261
>> |2,100,21,7815.2^X.X.X.13^Port 57377
>> 22:37:29.115 |MediaTerminationPointControl(1)::logCapabilitiesinTrace --
>> Device Caps = 15 16 11 12 |2,100,21,7815.2^X.X.X.13^Port 57377
>> 22:37:29.115 |RegionsServer::MatchCapabilities -- kbps=8, capACount=4,
>> capBCount=9|*^*^*****
>>
>>
>> Here is the TCS and MasterSalveDetermination from the .13 host that you
>> see in the diagram above, but that's just too late, looks like this:****
>>
>> 05:37:29.426 |
>> H245ASN - TtPid=(5551) [0xb0edd038 3252 bytes] -Incoming #103060 -value
>> MultimediaSystemControlMessage ::= request : terminalCapabilitySet :
>> {
>> sequenceNumber 2,
>> protocolIdentifier { 0 0 8 245 0 10 },
>> multiplexCapability h2250Capability :
>> {
>> maximumAudioDelayJitter 60,
>> receiveMultipointCapability
>> {
>> multicastCapability FALSE,
>> multiUniCastConference FALSE,
>> mediaDistributionCapability
>> {
>> {
>> centralizedControl FALSE,
>> distributedControl FALSE,
>> centralizedAudio FALSE,
>> distributedAudio FALSE,
>> centralizedVideo FALSE,
>> distributedVideo FALSE
>> }
>> }
>> },
>> transmitMultipointCapability
>> {
>> multicastCapability FALSE,
>> multiUniCastConference FALSE,
>> mediaDistributionCapability
>> {
>> {
>> centralizedControl FALSE,
>> distributedControl FALSE,
>> centralizedAudio FALSE,
>> distributedAudio FALSE,
>> centralizedVideo FALSE,
>> distributedVideo FALSE
>> }
>> }
>> },
>> receiveAndTransmitMultipointCapability
>> {
>> multicastCapability FALSE,
>> multiUniCastConference FALSE,
>> mediaDistributionCapability
>> {
>> {
>> centralizedControl FALSE,
>> distributedControl FALSE,
>> centralizedAudio FALSE,
>> distributedAudio FALSE,
>> centralizedVideo TRUE,
>> distributedVideo FALSE
>> }
>> }
>> },
>> mcCapability
>> {
>> centralizedConferenceMC FALSE,
>> decentralizedConferenceMC FALSE
>> },
>> rtcpVideoControlCapability FALSE,
>> mediaPacketizationCapability
>> {
>> h261aVideoPacketization FALSE
>> },
>> logicalChannelSwitchingCapability FALSE,
>> t120DynamicPortCapability FALSE
>> },
>> capabilityTable
>> {
>> {
>> capabilityTableEntryNumber 1,
>> capability nonStandard :
>> {
>> nonStandardIdentifier h221NonStandard :
>> {
>> t35CountryCode 181,
>> t35Extension 0,
>> manufacturerCode 18
>> },
>> data '01000000140000001000E0B7C0EE86B6'H
>> }
>> },
>> {
>> capabilityTableEntryNumber 2,
>> capability genericControlCapability :
>> {
>> capabilityIdentifier standard : { 0 0 8 323 1 3 1 }
>> }
>> },
>> {
>> capabilityTableEntryNumber 3,
>> capability receiveAudioCapability : g729AnnexAwAnnexB : 6
>> },
>> {
>> capabilityTableEntryNumber 4,
>> capability receiveAudioCapability : g729 : 6
>> },
>> {
>> capabilityTableEntryNumber 5,
>> capability receiveAudioCapability : g729AnnexA : 6
>> },
>> {
>> capabilityTableEntryNumber 6,
>> capability receiveAndTransmitUserInputCapability : dtmf : NULL
>> },
>> {
>> capabilityTableEntryNumber 7,
>> capability receiveAndTransmitUserInputCapability : basicString
>> : NULL
>> },
>> {
>> capabilityTableEntryNumber 44,
>> capability receiveAndTransmitUserInputCapability : hookflash :
>> NULL
>> }
>> },
>> capabilityDescriptors
>> {
>> {
>> capabilityDescriptorNumber 0,
>> simultaneousCapabilities
>> {
>> {
>> 3,
>> 4,
>> 5
>> |*^*^*
>> 05:37:29.427 |
>> H245ASN - TtPid=(5551) [0xb0e99b58 1444 bytes] -Incoming #103061 -value
>> MultimediaSystemControlMessage ::= request : masterSlaveDetermination :
>> {
>> terminalType 60,
>> statusDeterminationNumber 7594376
>> }
>> |*^*^*****
>>
>> ** **
>>
>> Which from my experience, would tell me that the .13 host would have won
>> the MasterSlaveDetermination, making the codec of choice be g711 and
>> forcing the .9 host to resolve the mismatch with a transcoder. However,
>> each side tried to fix the codec mismatch, all before
>> MasterSlaveDetermination even happened.****
>>
>> Did each side jump the gun, and begin invoking the transcoders too
>> early? Are the logs out of sequence from what actually happened on the
>> wire? Why would both sides attempt to resolve the codec mismatch?****
>>
>> For comparison, here is the same message flow from the perspective of the
>> .13 host logs.****
>>
>> <image002.png>****
>>
>> ** **
>>
>> And in summary, I know what is needed to fix this: fix the region
>> relationship from the transcoder to the g711 device so that it's 64kbps.
>>
>> I would like to understand what determines which side sends and
>> subsequently wins MasterSlaveDetermination on ICTs? I know it's based on
>> terminalType value and which is higher, and then a tie breaker is the
>> statusDeterminationNumber value. Is this a random election? Is it
>> controllable? I.e., Can I adjust the terminalType such that it forces
>> Master or Slave by setting a very High or Low value respectively?****
>>
>> If you've made it this far in my email, thank you! I realize I have
>> questions littered throughout. It's just the casual tone of my email. If
>> the email doesn't flow very eloquently, I do apologize. It's long and I
>> was going back and forth making edits to different sections.
>>
>> Please let me know if you need more details. Thank you for your time.***
>> *
>>
>>
>>
>> itevomcid ****
>>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20130918/0a6201db/attachment.html>
More information about the cisco-voip
mailing list