[cisco-voip] CUCM Upgrade failure...
Jonathan Charles
jonvoip at gmail.com
Tue Nov 17 14:02:40 EST 2015
admin:show hardware
HW Platform : 7845I2
Processors : 2
Type : Family: Xeon
CPU Speed : 2333
Memory : 8192 MBytes
Object ID : 1.3.6.1.4.1.9.1.587
OS Version : UCOS 5.0.0.0-2
Serial Number : KXXXXX
RAID Version :
RAID Firmware Version: 5.2-0 (16002)
RAID BIOS Version: 5.2-0 (16002)
BIOS Information :
1.18
RAID Details :
Controllers found: 1
----------------------------------------------------------------------
Controller information
----------------------------------------------------------------------
Controller Status : Okay
Channel description : SAS/SATA
Controller Model : IBM ServeRAID 8k
Controller Serial Number : ************
Physical Slot : 0
Copyback : Disabled
Data scrubbing : Enabled
Defunct disk drive count : 0
Logical drives/Offline/Critical : 0/0/0
---------------------------------------------------
Controller Version Information
---------------------------------------------------
BIOS : 5.2-0 (16002)
Firmware : 5.2-0 (16002)
Driver : 1.1-5 (24702)
Boot Flash : 5.1-0 (15429)
---------------------------------------------------
Controller Battery Information
---------------------------------------------------
Status : Okay
Over temperature : No
Capacity remaining : 100 percent
Time remaining (at current draw) : 2 days, 10 hours, 40 minutes
---------------------------------------------------
Controller Vital Product Data
---------------------------------------------------
VPD Assigned# : 39R8875
EC Version# : J85096
Controller FRU# : 25R8076
Battery FRU# : 25R8088
----------------------------------------------------------------------
Logical drive information
----------------------------------------------------------------------
No logical drives configured
----------------------------------------------------------------------
Physical drive information
----------------------------------------------------------------------
Device #0
Device is a Hard drive
State : Ready
Transfer Speed : SAS 3.0 Gb/s
Reported Channel,Device : 0,0
Vendor : IBM-ESXS
Model : CBRCA146C3ETS0 N
Firmware : C370
Serial number : PXXXXXX
World-wide name : 5000CCA00A45438B
Size : 140013 MB
Write Cache : Disabled (write-through)
FRU : 42D0422
PFA : No
Device #1
Device is a Hard drive
State : Ready
Transfer Speed : SAS 3.0 Gb/s
Reported Channel,Device : 0,1
Vendor : IBM-ESXS
Model : CBRCA146C3ETS0 N
Firmware : C370
Serial number : PXXXXXX
World-wide name : 5000CCA00A47B6CF
Size : 140013 MB
Write Cache : Disabled (write-through)
FRU : 42D0422
PFA : No
Device #2
Device is a Hard drive
State : Ready
Transfer Speed : SAS 3.0 Gb/s
Reported Channel,Device : 0,2
Vendor : IBM-ESXS
Model : CBRCA146C3ETS0 N
Firmware : C370
Serial number : PCSXXXXE
World-wide name : 5000CCA00A487C57
Size : 140013 MB
Write Cache : Disabled (write-through)
FRU : 42D0422
PFA : No
Device #3
Device is a Hard drive
State : Ready
Transfer Speed : SAS 3.0 Gb/s
Reported Channel,Device : 0,3
Vendor : IBM-ESXS
Model : CBRCA146C3ETS0 N
Firmware : C370
Serial number : PXXXXX
World-wide name : 5000CCA00A454387
Size : 140013 MB
Write Cache : Disabled (write-through)
FRU : 42D0422
PFA : No
Device #4
Device is a Enclosure
State : Standby
Transfer Speed : SAS 3.0 Gb/s
Reported Channel,Device : 2,0
Vendor : IBM-ESXS
Model : VSC7160
Firmware : 1.07
Command completed successfully.
Controllers found: 1
Current operation : None
Command completed successfully.
admin:
On Tue, Nov 17, 2015 at 12:36 PM, Ryan Huff <ryanhuff at outlook.com> wrote:
> I apologize, that should be "show hardware".
>
>
> -Ryan
> ------------------------------
> *From:* cisco-voip <cisco-voip-bounces at puck.nether.net> on behalf of Ryan
> Huff <ryanhuff at outlook.com>
> *Sent:* Tuesday, November 17, 2015 1:24:30 PM
> *To:* Jonathan Charles
>
> *Cc:* cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] CUCM Upgrade failure...
>
>
> Jonathan,
>
>
> Is this an MCS 78xx-I3? If it is and it has the LSI RAID Controller it
> will run a battery learning cycle every 2 months. During the cycle,
> writeback cache is disabled to keep the I/O from spiking. Ordinarily, the
> write-back cache should be re-enabled once the learning cycle is over.
>
>
> Can you show me the results of the command, "show hardware status" from
> the CLI of the pub?
>
>
> -Ryan
>
> ------------------------------
> *From:* Jonathan Charles <jonvoip at gmail.com>
> *Sent:* Tuesday, November 17, 2015 1:00 PM
> *To:* Ryan Huff
> *Cc:* Dave Goodwin; cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] CUCM Upgrade failure...
>
> Well the battery looks fine:
>
> [image: Inline image 1]
>
> But the write cache is disabled...
>
> [image: Inline image 2]
>
>
> I wonder if that is causing the time out...
>
>
>
> Jonathan
>
>
>
> On Tue, Nov 17, 2015 at 11:58 AM, Ryan Huff <ryanhuff at outlook.com> wrote:
>
>> Just as a note of practice; I wouldn't do anything to a running CCM at
>> the OS level during production, at least not without proper maintenance
>> declaration to the business unit [image: 😊]. "*A maintenance window a
>> day, keeps the resumes away*" .... lol.
>>
>>
>> If you've got RAID issues; I would get that solved before doing anything
>> else. If DRF services are still working (50/50 chance) I would get a good
>> DRS on that cluster *ASAP*. If DRS doesn't work, hopefully BAT still
>> does, I would do a BAT export on everything ASAP.
>>
>>
>> An upgrade would be the least of my worries at this point. He is a
>> reference to the particular error message you show in the image.(
>> http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucm/upgrade/9_1_1/CUCM_BK_UAEC4331_00_upgrade-guide-cucm-91/CUCM_BK_UAEC4331_00_upgrade-guide-cucm-91_chapter_0110.html#CUP0_RF_WD01249D_00
>> ).
>>
>>
>> -Ryan
>>
>> ------------------------------
>> *From:* Jonathan Charles <jonvoip at gmail.com>
>> *Sent:* Tuesday, November 17, 2015 12:45 PM
>> *To:* Dave Goodwin
>> *Cc:* Ryan Huff; cisco-voip at puck.nether.net
>>
>> *Subject:* Re: [cisco-voip] CUCM Upgrade failure...
>>
>> I did notice this on the Publisher OS Admin - Install/Upgrade page:
>>
>> [image: Inline image 1]
>>
>> I wonder if that's causing it...
>>
>>
>>
>> Jonathan
>>
>> On Tue, Nov 17, 2015 at 11:40 AM, Jonathan Charles <jonvoip at gmail.com>
>> wrote:
>>
>>> Yep, didn't work... same error. Let me check the other nodes...
>>>
>>> On Tue, Nov 17, 2015 at 11:34 AM, Jonathan Charles <jonvoip at gmail.com>
>>> wrote:
>>>
>>>> Logged into OS Admin on the pub, and it didn't give me the option to
>>>> assume control... can I run discovery/migration during production? Just
>>>> want to make sure I don't pop anything....
>>>>
>>>>
>>>>
>>>> Jonathan
>>>>
>>>> On Tue, Nov 17, 2015 at 11:30 AM, Dave Goodwin <
>>>> dave.goodwin at december.net> wrote:
>>>>
>>>>> I had a similar symptom a couple months ago - discovery timed out on
>>>>> one or more nodes. I found a supportforum post from another individual
>>>>> having that issue, and it appeared that PCD got "stuck" installing
>>>>> the ciscocm.ucmap_platformconfig.cop file in the middle. The way to work
>>>>> around it was to log into the Software Install/Upgrade page on the affected
>>>>> node(s) and you can see you'll have the option to Assume Control of a
>>>>> currently running install. Do that, and click through to complete the
>>>>> install. Then try to do the discovery again. Unknown if there was an
>>>>> existing bug causing this problem, but the above workaround steps solved my
>>>>> problem.
>>>>>
>>>>> On Tue, Nov 17, 2015 at 12:25 PM, Jonathan Charles <jonvoip at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I just posted the log... I am not sure from reading it...
>>>>>>
>>>>>> discovery node 1617 failed with errorcode 1-3
>>>>>>
>>>>>>
>>>>>> Jonathan
>>>>>>
>>>>>> On Tue, Nov 17, 2015 at 11:23 AM, Ryan Huff <ryanhuff at outlook.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Jonathan,
>>>>>>>
>>>>>>> What timesout on the publisher? Are you referencing when PCD tries
>>>>>>> to do the cluster discovery on the existing cluster?
>>>>>>>
>>>>>>> -Ryan
>>>>>>>
>>>>>>>
>>>>>>> Sent from my T-Mobile 4G LTE Device
>>>>>>>
>>>>>>>
>>>>>>> -------- Original message --------
>>>>>>> From: Jonathan Charles
>>>>>>> Date:11/17/2015 12:06 PM (GMT-05:00)
>>>>>>> To: Anthony Holloway
>>>>>>> Cc: cisco-voip at puck.nether.net
>>>>>>> Subject: Re: [cisco-voip] CUCM Upgrade failure...
>>>>>>>
>>>>>>> Nope, not all good... new error... the upgrade failed so I deleted
>>>>>>> the cluster and re-added... it finds all of the subscribers, but it times
>>>>>>> out on the publisher....
>>>>>>>
>>>>>>> I have verified all services are running on the Pub and it looks
>>>>>>> clean...
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Jonathan
>>>>>>>
>>>>>>> On Mon, Nov 16, 2015 at 10:20 AM, Anthony Holloway <
>>>>>>> avholloway+cisco-voip at gmail.com> wrote:
>>>>>>>
>>>>>>>> Looks like you're all good now, but as a heads up to everyone else,
>>>>>>>> don't stop at checking NTP with "utils ntp status". You will fail to
>>>>>>>> upgrade if your NTP configuration has an FQHN for the NTP server which
>>>>>>>> begins with a digit.
>>>>>>>>
>>>>>>>> E.g., 0.pool.ntp.org
>>>>>>>>
>>>>>>>> You will not see the hostname in the output of "utils ntp status",
>>>>>>>> as it will only show you the resolved IP address. So, you will also need
>>>>>>>> to issue a "utils ntp config" to see what value was entered by the
>>>>>>>> administrator.
>>>>>>>>
>>>>>>>> This is the only defect reference I found, though my upgrade I hit
>>>>>>>> it on was an 8.6 to 10.5 Refresh Upgrade (RU) (Not PCD).
>>>>>>>>
>>>>>>>> https://tools.cisco.com/bugsearch/bug/CSCtj07817
>>>>>>>>
>>>>>>>> On Sun, Nov 15, 2015 at 10:43 PM, Jonathan Charles <
>>>>>>>> jonvoip at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> OK, a reboot of CPCD got it passed that error...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jonathan
>>>>>>>>>
>>>>>>>>> On Sun, Nov 15, 2015 at 9:40 PM, Jonathan Charles <
>>>>>>>>> jonvoip at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yeah, the error I am getting is:
>>>>>>>>>>
>>>>>>>>>> 1 nodes(s) in Export task action ID #1127... on the Publisher...
>>>>>>>>>>
>>>>>>>>>> I will try rebooting everything...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Jonathan
>>>>>>>>>>
>>>>>>>>>> On Sun, Nov 15, 2015 at 9:35 PM, Ryan Huff <ryanhuff at outlook.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Looks healthy ...
>>>>>>>>>>>
>>>>>>>>>>> I recall trying PCD once and I hit really strange issues too.
>>>>>>>>>>> For that upgrade, I ultimately abandoned PCD and built new VMs with the
>>>>>>>>>>> Answer File Generator then a DRS backup/restore.
>>>>>>>>>>>
>>>>>>>>>>> Not sure where you are in your timeline or if it is that
>>>>>>>>>>> important but it is definitely something I would consider. Sometimes you
>>>>>>>>>>> can spend more time trying to get the silly tools to work, than to just do
>>>>>>>>>>> the work yourself.
>>>>>>>>>>>
>>>>>>>>>>> Google is littered with PCD weirdness; great idea of an
>>>>>>>>>>> application, just not there yet IMO.
>>>>>>>>>>>
>>>>>>>>>>> -Ryan
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Sent from my iPad
>>>>>>>>>>> On Nov 15, 2015, at 10:19 PM, Jonathan Charles <
>>>>>>>>>>> jonvoip at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Everything looks good....
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> admin:utils ntp status
>>>>>>>>>>> ntpd (pid 19674) is running...
>>>>>>>>>>>
>>>>>>>>>>> remote refid st t when poll reach delay
>>>>>>>>>>> offset jitter
>>>>>>>>>>>
>>>>>>>>>>> ==============================================================================
>>>>>>>>>>> 127.127.1.0 .LOCL. 10 l 21 64 377 0.000
>>>>>>>>>>> 0.000 0.001
>>>>>>>>>>> 10.0.31.2 10.0.31.3 3 u 175 1024 377 0.635
>>>>>>>>>>> -5.771 2.042
>>>>>>>>>>> *10.0.31.3 129.6.15.29 2 u 970 1024 377 0.510
>>>>>>>>>>> -11.340 0.449
>>>>>>>>>>> 10.1.31.2 10.0.31.3 3 u 490 1024 377 0.850
>>>>>>>>>>> -9.114 4.881
>>>>>>>>>>> +10.1.31.3 129.6.15.29 2 u 184 1024 377 0.817
>>>>>>>>>>> -4.085 5.355
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> synchronised to NTP server (10.0.31.3) at stratum 3
>>>>>>>>>>> time correct to within 68 ms
>>>>>>>>>>> polling server every 1024 s
>>>>>>>>>>>
>>>>>>>>>>> Current time in UTC is : Mon Nov 16 03:16:14 UTC 2015
>>>>>>>>>>> Current time in America/Chicago is : Sun Nov 15 21:16:14 CST 2015
>>>>>>>>>>> admin:
>>>>>>>>>>>
>>>>>>>>>>> admin:utils diagnose module validate_network
>>>>>>>>>>>
>>>>>>>>>>> Log file: platform/log/diag1.log
>>>>>>>>>>>
>>>>>>>>>>> Starting diagnostic test(s)
>>>>>>>>>>> ===========================
>>>>>>>>>>> test - validate_network : Passed
>>>>>>>>>>>
>>>>>>>>>>> Diagnostics Completed
>>>>>>>>>>>
>>>>>>>>>>> admin:#
>>>>>>>>>>>
>>>>>>>>>>> admin:utils dbreplication runtimestate
>>>>>>>>>>>
>>>>>>>>>>> DB and Replication Services: ALL RUNNING
>>>>>>>>>>>
>>>>>>>>>>> Cluster Replication State: Replication repair command started
>>>>>>>>>>> at: 2014-06-20-23-22
>>>>>>>>>>> Replication repair command COMPLETED 541 tables processed
>>>>>>>>>>> out of 541
>>>>>>>>>>> Errors or Mismatches Were Found:
>>>>>>>>>>>
>>>>>>>>>>> Use 'file view activelog
>>>>>>>>>>> cm/trace/dbl/sdi/ReplicationRepair.2014_06_20_23_22_51.out' to see the
>>>>>>>>>>> details
>>>>>>>>>>>
>>>>>>>>>>> DB Version: ccm8_6_2_20000_2
>>>>>>>>>>> Number of replicated tables: 541
>>>>>>>>>>>
>>>>>>>>>>> Cluster Detailed View from PUB (5 Servers):
>>>>>>>>>>>
>>>>>>>>>>> PING REPLICATION
>>>>>>>>>>> REPL. DBver& REPL. REPLICATION SETUP
>>>>>>>>>>> SERVER-NAME IP ADDRESS (msec) RPC? STATUS
>>>>>>>>>>> QUEUE TABLES LOOP? (RTMT) & details
>>>>>>>>>>> ----------- ------------ ------ ---- -----------
>>>>>>>>>>> ----- ------- ----- -----------------
>>>>>>>>>>> IPTCMS02 10.0.126.12 0.196 Yes Connected 0
>>>>>>>>>>> match Yes (2) Setup Completed
>>>>>>>>>>> IPTCMS01 10.0.126.11 0.151 Yes Connected 0
>>>>>>>>>>> match Yes (2) Setup Completed
>>>>>>>>>>> IPTCMP 10.0.126.10 0.065 Yes Connected 0
>>>>>>>>>>> match Yes (2) PUB Setup Completed
>>>>>>>>>>> IPTCMS03 10.1.126.13 0.545 Yes Connected 0
>>>>>>>>>>> match Yes (2) Setup Completed
>>>>>>>>>>> IPTCMS04 10.1.126.14 0.527 Yes Connected 0
>>>>>>>>>>> match Yes (2) Setup Completed
>>>>>>>>>>>
>>>>>>>>>>> admin:
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Nov 15, 2015 at 9:14 PM, Ryan Huff <ryanhuff at outlook.com
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Also worth noting that if CCM NTP is synchronized to a Windows
>>>>>>>>>>>> server (even if it shows Stratum 3 or better); that is a problem you'll
>>>>>>>>>>>> need to correct as SNTP can play hell with UCOS and do some pretty weird
>>>>>>>>>>>> stuff.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>>
>>>>>>>>>>>> On Nov 15, 2015, at 10:06 PM, Ryan Huff <ryanhuff at outlook.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> If the FROM CCM version was unrestricted, it would say "
>>>>>>>>>>>> *Unrestricted*" after the version number on the "*Active
>>>>>>>>>>>> Master Version"* line. If it does not say "*Unrestricted*",
>>>>>>>>>>>> then it is the more common restricted version.
>>>>>>>>>>>>
>>>>>>>>>>>> As to your original issue, I would start with all the usual
>>>>>>>>>>>> suspects. Is the FROM CCM cluster healthy to start with; dns, ntp,
>>>>>>>>>>>> replication ... etc?
>>>>>>>>>>>>
>>>>>>>>>>>> From CCM:
>>>>>>>>>>>>
>>>>>>>>>>>> 1.) #utils diagnose module validate_network
>>>>>>>>>>>> (Should see *Passed*)
>>>>>>>>>>>>
>>>>>>>>>>>> 2.) #utils ntp status
>>>>>>>>>>>> (Pub should be synchronized and Strata 3 or better)
>>>>>>>>>>>>
>>>>>>>>>>>> 3.) #utils dbreplication runtimestate
>>>>>>>>>>>> (Should see *2 - Setup Completed* for all nodes)
>>>>>>>>>>>>
>>>>>>>>>>>> If PCD is moving the apps to a new platform/chassis, make sure
>>>>>>>>>>>> the *target* environment can reach all the same network assets
>>>>>>>>>>>> as the *from* environment.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> Ryan
>>>>>>>>>>>>
>>>>>>>>>>>> On Nov 15, 2015, at 9:34 PM, Jonathan Charles <
>>>>>>>>>>>> jonvoip at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> It just says the version number...
>>>>>>>>>>>>
>>>>>>>>>>>> admin:show version active
>>>>>>>>>>>> Active Master Version: 8.6.2.20000-2
>>>>>>>>>>>> Active Version Installed Software Options:
>>>>>>>>>>>> cmterm-7942_7962-sccp.9-3-1ES27-rel.cop
>>>>>>>>>>>> cmterm-devicepack8.6.2.24118-1.cop
>>>>>>>>>>>> ciscocm.refresh_upgrade_v1.1.cop
>>>>>>>>>>>> ciscocm.ucmap_platformconfig.cop
>>>>>>>>>>>> ciscocm.migrate-export-v1.12.cop
>>>>>>>>>>>> admin:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Jonahan
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Nov 15, 2015 at 7:26 PM, Ryan Huff <
>>>>>>>>>>>> ryanhuff at outlook.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Did you actually dump the logs to a serial interface (curious
>>>>>>>>>>>>> what it shows)?
>>>>>>>>>>>>>
>>>>>>>>>>>>> On the FROM CCM, goto the CLI of pub (or sub) and do a "show
>>>>>>>>>>>>> version active"; it will tell you if you have the UNREST.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ryan
>>>>>>>>>>>>>
>>>>>>>>>>>>> > On Nov 15, 2015, at 7:55 PM, Jonathan Charles <
>>>>>>>>>>>>> jonvoip at gmail.com> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Using PCD on CUCM 10.5.2.11901 got the following error:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > <image.png>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > It seems to imply I am not matching restricted vs.
>>>>>>>>>>>>> unrestricted...
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Any easy way to find out?
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Jonathan
>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>> > cisco-voip mailing list
>>>>>>>>>>>>> > cisco-voip at puck.nether.net
>>>>>>>>>>>>> > https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> cisco-voip mailing list
>>>>>>>>>>>> cisco-voip at puck.nether.net
>>>>>>>>>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> cisco-voip mailing list
>>>>>>>>> cisco-voip at puck.nether.net
>>>>>>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> cisco-voip mailing list
>>>>>> cisco-voip at puck.nether.net
>>>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/b06d7ac4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 12690 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/b06d7ac4/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 42074 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/b06d7ac4/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 16236 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/b06d7ac4/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OutlookEmoji-?.png
Type: image/png
Size: 488 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20151117/b06d7ac4/attachment-0003.png>
More information about the cisco-voip
mailing list