<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>In general Raid1 should allow the system to boot off of a single drive after a hardware failure. That's what it is designed to do (afaik, I'm not a raid expert only play one on the internet). On the 7825 array in my experience if you are doing lots of pulling drives, failing drives, upgrading firmware on drives, etc that's going to mean lots and lots of array resyncing. If that resynchronization doesn't complete you don't have a redundant array. The server won't be able to use the backup drive if the failure occurs when the array isn't optimal.</div><div><br></div>More inline below...<div><br><div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div>-Ryan</div></span>
</div><blockquote type="cite">
<br><div><div>On Feb 14, 2012, at 10:43 AM, Sinisa Djokic wrote:</div><br class="Apple-interchange-newline"><span class="Apple-style-span" style="border-collapse: separate; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; font-family: 'Lucida Grande'; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">hi group..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">does anybody have expertise on this matter..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">maybe wes or ryan have inside info..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">we had a TAC case with failed HDD and problematic RAID controller in MCS-7825-I4 server running CUCM 7.1.5..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">the symptom happend when we shutdown the server in order to check what IBR FRU is on memory inside the server..we checked it and after that we powered up the server.. RAID controller reports DEGRADED state..cisco OS started to boot and in certain point it reported something like “ unsupported hardware......not for the production..without TAC support..bla..bla..bla )”..we opened the case..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">during the case we upgraded all kinds of firmware on server and also find out that one HDD failed..we replaced it and troubleshooted more..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">finally we got RMA for the server but have spotted very strange behavior on the new server as well ( we put in into lab to test it )..so, we’re thinking , is the following expected behaviour or there is a serious problem in cisco OS or IBM HW..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><span>1.<font size="1" face="Times New Roman"><span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></font></span></span></font>first of all we upgraded the HDD firmware on the new server we got from RMA to 3B06 ( it was with 3B05 ) having in mind<span class="Apple-converted-space"> </span><b><span style="font-weight: bold; ">CSCti52867<span class="Apple-converted-space"> </span></span></b>which we hitted earlier..after that we’installed CUCM 7.1.5..</div></div></div></span></div></blockquote><div>At this point I'd guess that even 3B06 is old. Make sure your drives are on the latest firmware, and make sure your array is up and completely stable before applying the update. If it's degraded when you try to patch it can miss one of the drives and leave it on old firmware. If you hit CSCti52867 then you absolutely have to run the diskex...cop.sgn file (when the array is optimal) and really should rebuild the server. Just upgrading the firmware is not sufficient to fix or prevent future outages once the drives have started failing due to this bug.</div><br><blockquote type="cite"><div><span class="Apple-style-span" style="border-collapse: separate; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; font-family: 'Lucida Grande'; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><o:p></o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><span>2.<font size="1" face="Times New Roman"><span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></font></span></span></font>when server is shut down and we pull out the right hard drive ( the one in BAY 1)..we power up the server..RAID controller detects one drive missing and reports DEGRADED state..cisco OS starts to boot and in certain point it reports again something like “ unsupported hardware......not for the production..without TAC support..bla..bla..bla )..</div></div></div></span></div></blockquote><div>See the bug Wes pointed out. This is not expected behavior.</div><div><br></div><blockquote type="cite"><div><span class="Apple-style-span" style="border-collapse: separate; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; font-family: 'Lucida Grande'; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><o:p></o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><span>3.<font size="1" face="Times New Roman"><span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></font></span></span></font>when we shut down the server again..and switch the scenario..we get back the right HDD in server ( BAY 1 ) and pull out the left HDD out ( BAY 0 )..when we power up the server..it doesn’t even detect bootable device like there is no HDDs inside..</div></div></div></span></div></blockquote><div>If the BIOS doesn't detect the drive you've got a hardware issue either with the array or the disk, period. If the BIOS detects the drive but no OS is found then the drive most likely didn't get fully sync'd from a previous degraded state.</div><br><blockquote type="cite"><div><span class="Apple-style-span" style="border-collapse: separate; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; font-family: 'Lucida Grande'; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><o:p></o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><span>4.<font size="1" face="Times New Roman"><span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></font></span></span></font>so, when we shut down the server again..and get back both HDDs where they belong..power up the server..everythings works fine..of course, RAID is resynching..</div></div></div></span></div></blockquote><div>Does it ever finish resyncing? That can take hours and until it does the drive that is not in a good state is not redundant. </div><br><blockquote type="cite"><div><span class="Apple-style-span" style="border-collapse: separate; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; font-family: 'Lucida Grande'; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><o:p></o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><span>5.<font size="1" face="Times New Roman"><span style="font: normal normal normal 7pt/normal 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></font></span></span></font>of course both HDDs used in server are in correct state and aren’t failed..</div></div></div></span></div></blockquote><div>I wouldn't place bets on that just yet.</div><br><blockquote type="cite"><div><span class="Apple-style-span" style="border-collapse: separate; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; font-family: 'Lucida Grande'; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0.5in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; text-indent: -0.25in; "><o:p></o:p></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">so, i’m a little bit confused, since it’s happening on 2 different servers..identical scenario..my RAID1 perception is in serious doubt..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">does this mean that if HDD failed during regular maintenance shutdown ( which we had in the first place ), RAID shouldn’t provide operational system?..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">to be honest , we didn’t have failed disk while server was up and online so we couldn’t notice what’s going on in that situation..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">but the fact is, with the old server and with the new server, when we pull out one of the HDDs the system wouldn’t boot up properly..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">is this expected behaviour?..what’s the purpose of RAID1 then?..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">is RAID1 expected to cover just online HDD failure ( if so ) or it should work in every scenario?..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">any thoughts?..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><o:p> </o:p></span></font></div></div></div></span></div></blockquote><div><span class="Apple-style-span" style="border-collapse: separate; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; font-family: 'Lucida Grande'; "><div lang="EN-US" link="blue" vlink="purple"><div class="WordSection1" style="page: WordSection1; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">thanx..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; ">regards..<o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><o:p> </o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><b><font size="2" color="black" face="Arial"><span style="font-size: 10pt; font-family: Arial, sans-serif; color: black; font-weight: bold; ">Sinisa Djokic</span></font></b><font size="2" color="black" face="Arial"><span style="font-size: 10pt; font-family: Arial, sans-serif; color: black; "></span></font><font size="3" color="black" face="Arial"><span style="font-size: 12pt; font-family: Arial, sans-serif; color: black; "><o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" color="black" face="Arial"><span style="font-size: 10pt; font-family: Arial, sans-serif; color: black; ">System Engineer</span></font><font size="3" color="black" face="Arial"><span style="font-size: 12pt; font-family: Arial, sans-serif; color: black; "><br></span></font><font size="2" color="black" face="Arial"><span style="font-size: 10pt; font-family: Arial, sans-serif; color: black; ">CCIE #25996 Voice</span></font><font size="3" color="black" face="Arial"><span style="font-size: 12pt; font-family: Arial, sans-serif; color: black; "><o:p></o:p></span></font></div><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="3" face="Times New Roman"><span style="font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></span></font></div><table class="MsoNormalTable" border="0" cellpadding="0"><tbody><tr><td style="padding-top: 0.75pt; padding-right: 0.75pt; padding-bottom: 0.75pt; padding-left: 0.75pt; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="3" face="Times New Roman"><span style="font-size: 12pt; font-family: 'Times New Roman', serif; "><span><image002.png></span><o:p></o:p></span></font></div></td><td style="padding-top: 0.75pt; padding-right: 0.75pt; padding-bottom: 0.75pt; padding-left: 0.75pt; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="3" face="Times New Roman"><span style="font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></span></font></div></td><td style="padding-top: 0.75pt; padding-right: 0.75pt; padding-bottom: 0.75pt; padding-left: 0.75pt; "><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="1" color="black" face="Arial"><span style="font-size: 9pt; font-family: Arial, sans-serif; color: black; ">MDS Informaticki inzenjering<br>Milutina Milankovica 7d<br>11070 Novi Beograd, Serbia<br>Tel: +381 11 2015 200 +381 11 2015 200 , 2015 273<br>Fax: +381 11 3194 954</span></font><font size="1" face="Times New Roman"><span style="font-size: 9pt; font-family: 'Times New Roman', serif; "><br></span></font><font size="1" face="Arial"><span style="font-size: 9pt; font-family: Arial, sans-serif; "><a href="http://www.mds.rs/" style="color: blue; text-decoration: underline; "><font color="blue"><span style="color: blue; ">www.mds.rs</span></font></a></span></font><font size="1" face="Times New Roman"><span style="font-size: 9pt; font-family: 'Times New Roman', serif; "><br></span></font><font size="1" face="Arial"><span style="font-size: 9pt; font-family: Arial, sans-serif; "><a href="mailto:sdjokic@mds.rs" style="color: blue; text-decoration: underline; "><font color="blue"><span style="color: blue; ">sdjokic@mds.rs</span></font></a></span></font><font size="1" face="Times New Roman"><span style="font-size: 9pt; font-family: 'Times New Roman', serif; "><o:p></o:p></span></font></div></td></tr></tbody></table><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="1" color="#a6a6a6" face="Arial"><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(166, 166, 166); ">This e-mail message and any attachment are intended exclusively for the named addressee. They may contain confidential information which may also be protected by professional secrecy. Unless you are the named addressee (or authorised to receive for the addressee) you may not copy or use this message or any attachment or disclose the contents to anyone else. If this e-mail was sent to you by mistake please notify the sender immediately and delete this e-mail.<o:p></o:p></span></font></div><p class="MsoNormal" style="margin-top: 6pt; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><b><font size="5" color="green" face="Webdings"><span style="font-size: 16pt; font-family: Webdings; color: green; font-weight: bold; ">P</span></font></b><font size="1" color="#00b050" face="Arial"><span style="font-size: 8pt; font-family: Arial, sans-serif; color: rgb(0, 176, 80); "><span class="Apple-converted-space"> </span></span></font><font size="1" color="#00b050" face="Arial"><span style="font-size: 7pt; font-family: Arial, sans-serif; color: rgb(0, 176, 80); ">Save a tree. Don’t print this e-mail unless it’s really necessary.<o:p></o:p></span></font></p><div style="margin-top: 0in; margin-right: 0in; margin-left: 0in; margin-bottom: 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif; "><font size="2" face="Calibri"><span style="font-size: 11pt; "><o:p> </o:p></span></font></div></div>_______________________________________________<br>cisco-voip mailing list<br><a href="mailto:cisco-voip@puck.nether.net" style="color: blue; text-decoration: underline; ">cisco-voip@puck.nether.net</a><br><a href="https://puck.nether.net/mailman/listinfo/cisco-voip" style="color: blue; text-decoration: underline; ">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br></div></span></div><br></div></body></html>