[cisco-voip] Security By Default and ITL (was Phones Not Getting Auth, Idle, Services URLS, etc)

Sat Aug 13 08:10:41 EDT 2011

I wanted to follow up on this thread. The "show itl" command was a key piece
of the troubleshooting process that let us track this down. After this
incident though I wanted to take some time to explain how ITL and Security
By Default worked, and also document the common troubleshooting steps I
used.

Hopefully this document will help you out in the future!

https://supportforums.cisco.com/docs/DOC-17679

Support Forums mangles pictures in the thumbnail view, but you should be
able to click on each diagram to full size it and get a better understanding
of what goes on in the background.

-Jason Burns

On Thu, Jun 30, 2011 at 5:31 PM, Wes Sisk <wsisk at cisco.com> wrote:

> **
> Summary for the innocent bystanders:
>
> * Phone previously registered to 7.1.5 cluster with no CTL.  Phone got ITL
> file from new cluster successfully.
> * Phone console logs show:
> 2155: ERR 12:08:23.841097 SECD: EROR:verifyFile: sgn verify file
> failed</usr/ram/SEP00260BD749E9.cnf.xml>, errclass 8, errcode 19 (signer not
> in CTL)
> 2156: ERR 12:08:23.841825 SECD: EROR:verifyFile: verify
> FAILED,</usr/ram/SEP00260BD749E9.cnf.xml>
> 2157: NOT 12:08:23.844821 SECD: sendRespToClient: Sent the response to the
> TVS client, len : 2056
> 2158: NOT 12:08:23.860569 tftpClient: authorize file = 13, isEncr = 0
>
> * phones show ITL downloading successfully. suspect problem with ITL file.
> Get TVS logs.
>
> * This is where things got sketchy. CLI command 'show ITL' shows the TFTP
> entry for the publisher with a serial number.  We checked all certificates
> in ccmadmin under system->security->certificates.  No serial numbers match
> the current ITL file.  We tried regenerating CallManager.pem as that should
> be the cert TFTP uses. After regenerating, restarting TVS, and restarting
> TFTP the cli command 'show itl' still shows incorrect serial number for TFTP
> certificate.  This means ITL file is not being updated properly.
>
> * Checked TVS logs.  Many red herrings there but one substantial error:
> 11:45:18.827 |   debug ERROR:writeFile () - Unable to open file
> /usr/local/cm/tftp/ITLFile.tlv for writing (errno - 13)
> 11:45:18.827 |   debug In function : tvsGetPublicKeyFromX509
> 11:45:19.100 |ITLFileRegenerated - New ITL File has been generated.
>
> TVS could not write new ITL file.
>
> Checked filesystem permissions on Matthew's box:
> [root at server ~]# ls -la /usr/local/cm/tftp/ITLFile.tlv
> -rw-r--r--  1 ctftp ccmbase 3664 May 25 13:18
> /usr/local/cm/tftp/ITLFile.tlv
>
> On lab box:
> [root at CUCM85Pub sdi]# ls -la /usr/local/cm/tftp/ITLFile.tlv
> -rw-r--r--  1 certbase ccmbase 4930 Jun 30 15:38
> /usr/local/cm/tftp/ITLFile.tlv
>
> Permissions issue.  On Matthew's box:
> [root at server ~]# ps -eaf | grep -iE "TFTP|TVS"
> certbase 31007 17132  0 15:35 ?        00:00:00 /usr/local/cm/bin/tvs
> ctftp    31574 17132  0 15:36 ?        00:00:02 /usr/local/cm/bin/ctftp
>
> Looks like ctftp service most likely wrote or created ITL file on May 25.
> The owner of ITLFile.tlv should be certbase not ctftp.  We did:
> 1. chown certbase ITLFile.tlv - this set the correct owner
> 2. restarted TVS service - TVS regenerated ITL file and successfully wrote
> it to the file system
> 3. restart TFTP service so it could pickup the new ITL file from the file
> system.
>
> After this phones successfully registered. So far there is at least one new
> bug out of this:
> CSCtr27100 TVS inaccurately reports New ITL File has been generated
>
> In CUCM 8.6 ctftp does indeed generate the ITL file.  In 8.5.1.11900-21
> (aka 8.5.1su1) that Matthew is running ITL file should be generated by TVS.
>
> Regards,
> Wes
>
>
> On 6/30/2011 4:17 PM, Matthew Loraditch wrote:
>
>  Well I just finished what amounted to 5 hours on the phone and 6 hours
> for Cisco. Most of that with Wes and apparently Ryan Ratliff and Jason Burns
> stopped by for a while as well!****
>
> Anyway we got it fixed. Wes said he is going to do a write up but the gist
> was the ITL cert was written by the wrong service and thus not matching up
>  and they had to go in with root and do something so that right service
> could properly create it.****
>
> ** **
>
> Major Kudos to Wes and if we are ever within 50 miles or less of each other
> drinks are in order!****
>
> ** **
>
> ** **
>
> *Matthew Loraditch, CCVP, CCNA, CCDA*
> 1965 Greenspring Drive****
>
> Timonium, MD 21093
> support at heliontechnologies.com
> (p) (410) 252-8830
> (F) (443) 541-1593
>
> Visit us at www.heliontechnologies.com
> Support Issue? Email support at heliontechnologies.com for fast assistance!**
> **
>
> ** **
>
> *From:* cisco-voip-bounces at puck.nether.net [
> mailto:cisco-voip-bounces at puck.nether.net<cisco-voip-bounces at puck.nether.net>]
> *On Behalf Of *Matthew Loraditch
> *Sent:* Thursday, June 30, 2011 11:24 AM
> *To:* Lelio Fulgenzi
> *Cc:* cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS,
> etc****
>
> ** **
>
> Wes is helping out and watching my case so far we have just verified normal
> settings, regenerated the tomcat certs to no avail and then totally factory
> defaulted the phone (the 123456789*0#) and now the phone comes up
> unprovisioned and won’t connect at all!****
>
> Engineers seem pretty stumped****
>
> ** **
>
> ** **
>
> *Matthew Loraditch, CCVP, CCNA, CCDA*
> 1965 Greenspring Drive****
>
> Timonium, MD 21093
> support at heliontechnologies.com
> (p) (410) 252-8830
> (F) (443) 541-1593
>
> Visit us at www.heliontechnologies.com
> Support Issue? Email support at heliontechnologies.com for fast assistance!**
> **
>
> ** **
>
> *From:* Lelio Fulgenzi [mailto:lelio at uoguelph.ca <lelio at uoguelph.ca>]
> *Sent:* Thursday, June 30, 2011 11:21 AM
> *To:* Matthew Loraditch
> *Cc:* Matthew Loraditch; cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS,
> etc****
>
> ** **
>
> Let us know how that goes...
>
> Sent from my iPhone****
>
>
> On Jun 30, 2011, at 9:09 AM, Matthew Loraditch <
> MLoraditch at heliontechnologies.com> wrote:****
>
>  Well I have done what Lelio suggested, I have rebooted the cluster. I
> noticed from the phone logs and in security settings that the certs they are
> getting say the hostname so I changed the CCMs from IP back to hostname.
> Still no dice…****
>
> Time to open a TAC case!****
>
>  ****
>
>  ****
>
> *Matthew Loraditch, CCVP, CCNA, CCDA*
> 1965 Greenspring Drive****
>
> Timonium, MD 21093
> support at heliontechnologies.com
> (p) (410) 252-8830
> (F) (443) 541-1593
>
> Visit us at www.heliontechnologies.com
> Support Issue? Email support at heliontechnologies.com for fast assistance!**
> **
>
>  ****
>
> *From:* cisco-voip-bounces at puck.nether.net [
> mailto:cisco-voip-bounces at puck.nether.net<cisco-voip-bounces at puck.nether.net>]
> *On Behalf Of *Matthew Loraditch
> *Sent:* Wednesday, June 29, 2011 9:57 PM
> *To:* Lelio Fulgenzi
> *Cc:* cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS,
> etc****
>
>  ****
>
> Will give it a whirl in the am****
>
>  ****
>
> *Matthew Loraditch, CCVP, CCNA, CCDA*
> 1965 Greenspring Drive****
>
> Timonium, MD 21093
> support at heliontechnologies.com
> (p) (410) 252-8830
> (F) (443) 541-1593
>
> Visit us at www.heliontechnologies.com
> Support Issue? Email support at heliontechnologies.com for fast assistance!**
> **
>
>  ****
>
> *From:* Lelio Fulgenzi [mailto:lelio at uoguelph.ca <lelio at uoguelph.ca>]
> *Sent:* Wednesday, June 29, 2011 9:43 PM
> *To:* Matthew Loraditch
> *Cc:* wsisk at cisco.com; cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS,
> etc****
>
>  ****
>
> Try the service parameter I mentioned. ****
>
>  ****
>
> If you do a search on the archives, Wes posted a link to the doc bug.
>
> Sent from my iPhone****
>
>
> On Jun 29, 2011, at 9:31 PM, Matthew Loraditch <
> MLoraditch at heliontechnologies.com> wrote:****
>
>  7942s and 7945s so far, in re someone else’s email have restarted tftp
> etc already.****
>
>  ****
>
>  ****
>
> *Matthew Loraditch, CCVP, CCNA, CCDA*
> 1965 Greenspring Drive****
>
> Timonium, MD 21093
> support at heliontechnologies.com
> (p) (410) 252-8830
> (F) (443) 541-1593
>
> Visit us at www.heliontechnologies.com
> Support Issue? Email support at heliontechnologies.com for fast assistance!**
> **
>
>  ****
>
> *From:* Lelio Fulgenzi [mailto:lelio at uoguelph.ca]
> *Sent:* Wednesday, June 29, 2011 7:50 PM
> *To:* Matthew Loraditch
> *Cc:* wsisk at cisco.com; cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS,
> etc****
>
>  ****
>
> Are these 7941/61? I had a problem where I had to change the service
> provisioning service parameter to external to make this work.
>
> Sent from my iPhone****
>
>
> On Jun 29, 2011, at 6:23 PM, Matthew Loraditch <
> MLoraditch at heliontechnologies.com> wrote:****
>
>  Googled and found that and it’s not working….
>
>
>
>
>
>
> Matthew Loraditch, CCVP, CCNA, CCDA
> 1965 Greenspring Drive
>
> Timonium, MD 21093
>
> support at heliontechnologies.com
> (p) (410) 252-8830
> (F) (443) 541-1593
>
> Visit us at www.heliontechnologies.com
>
> Support Issue? Email support at heliontechnologies.com
> for fast assistance!
>
>
>
>
>
>
>
>
> From: Wes Sisk
> [mailto:wsisk at cisco.com]
> Sent: Wednesday, June 29, 2011 5:24 PM
> To: Matthew Loraditch
> Cc: cisco-voip at puck.nether.net
> Subject: Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS, etc
>
>
>
>
>
> These lines pretty much say it all. verification of the downloaded file
> failed. Delete the CTL/ITL from the phone and try again.
> https://supportforums.cisco.com/docs/DOC-15799#Manual_ITL_Delete
>
> Regards,
> Wes
>
> On 6/29/2011 5:03 PM, Matthew Loraditch wrote:
>
> 1715: ERR 16:59:35.170584 SECD: EROR:verifyFile: sgn verify file failed
> </usr/ram/SEP00260BD749E9.cnf.xml>, errclass 8, errcode 19 (signer
> not in CTL)
>
> 1716: ERR 16:59:35.171327 SECD: EROR:verifyFile: verify FAILED,
> </usr/ram/SEP00260BD749E9.cnf.xml>
>
> Sent from my Android phone using TouchDown (www.nitrodesk.com) ****
>
>  _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip****
>
>
> _______________________________________________
> cisco-voip mailing listcisco-voip at puck.nether.nethttps://puck.nether.net/mailman/listinfo/cisco-voip
>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20110813/897491ce/attachment.html>