[cisco-voip] Security By Default and ITL (was Phones Not Getting Auth, Idle, Services URLS, etc)

Tue Aug 16 05:29:35 EDT 2011

There is a service parameter that allows you to revert back to an older
version.

Regards,

Ahmed Elnagar | Unified Communication Team Leader | CCIE #24697, Voice

Description: Description: MS Green

From: cisco-voip-bounces at puck.nether.net
[mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Casper, Steven
Sent: Monday, August 15, 2011 5:09 PM
To: 'Jason Burns'; Wes Sisk
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Security By Default and ITL (was Phones Not
Getting Auth, Idle, Services URLS, etc)

First of all this is an excellent thread with a lot of valuable details. I
have started to review upgrading from 7.1.5 to 8.5 or 8.6. Far as I can tell
all the Security by Default configuration should be automatically  done by
the upgrade process and once complete the TFTP servers should be reset and
the phones reset again to ensure the phones retrieve the ITL file. Seem
simple enough but the thought that something could go wrong and it could be
necessary to have users delete the ITL config from 12,000 phones is one that
could keep me up at night though.

How has it been going out there with Security by Default and upgrades to
8.x... any gotcha's or caveats? I am leaning towards the latest version of
8.5 over 8.6 since 8.6 does not seem to have many new features I need but is
one release preferred over the other based on the upgrade process? I am
running a multi-node cluster on 7.1.5.11900-6 in the non-secure mode

Steve

From: cisco-voip-bounces at puck.nether.net
[mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Jason Burns
Sent: Saturday, August 13, 2011 8:11 AM
To: Wes Sisk
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Security By Default and ITL (was Phones Not
Getting Auth, Idle, Services URLS, etc)

I wanted to follow up on this thread. The "show itl" command was a key piece
of the troubleshooting process that let us track this down. After this
incident though I wanted to take some time to explain how ITL and Security
By Default worked, and also document the common troubleshooting steps I
used.

Hopefully this document will help you out in the future!

https://supportforums.cisco.com/docs/DOC-17679

Support Forums mangles pictures in the thumbnail view, but you should be
able to click on each diagram to full size it and get a better understanding
of what goes on in the background.

-Jason Burns

On Thu, Jun 30, 2011 at 5:31 PM, Wes Sisk <wsisk at cisco.com> wrote:

Summary for the innocent bystanders:

* Phone previously registered to 7.1.5 cluster with no CTL.  Phone got ITL
file from new cluster successfully.
* Phone console logs show:
2155: ERR 12:08:23.841097 SECD: EROR:verifyFile: sgn verify file
failed</usr/ram/SEP00260BD749E9.cnf.xml>, errclass 8, errcode 19 (signer not
in CTL)
2156: ERR 12:08:23.841825 SECD: EROR:verifyFile: verify
FAILED,</usr/ram/SEP00260BD749E9.cnf.xml>
2157: NOT 12:08:23.844821 SECD: sendRespToClient: Sent the response to the
TVS client, len : 2056
2158: NOT 12:08:23.860569 tftpClient: authorize file = 13, isEncr = 0

* phones show ITL downloading successfully. suspect problem with ITL file.
Get TVS logs.

* This is where things got sketchy. CLI command 'show ITL' shows the TFTP
entry for the publisher with a serial number.  We checked all certificates
in ccmadmin under system->security->certificates.  No serial numbers match
the current ITL file.  We tried regenerating CallManager.pem as that should
be the cert TFTP uses. After regenerating, restarting TVS, and restarting
TFTP the cli command 'show itl' still shows incorrect serial number for TFTP
certificate.  This means ITL file is not being updated properly.

* Checked TVS logs.  Many red herrings there but one substantial error:
11:45:18.827 |   debug ERROR:writeFile () - Unable to open file
/usr/local/cm/tftp/ITLFile.tlv for writing (errno - 13)
11:45:18.827 |   debug In function : tvsGetPublicKeyFromX509
11:45:19.100 |ITLFileRegenerated - New ITL File has been generated. 

TVS could not write new ITL file.

Checked filesystem permissions on Matthew's box:
[root at server ~]# ls -la /usr/local/cm/tftp/ITLFile.tlv
-rw-r--r--  1 ctftp ccmbase 3664 May 25 13:18 /usr/local/cm/tftp/ITLFile.tlv

On lab box:
[root at CUCM85Pub sdi]# ls -la /usr/local/cm/tftp/ITLFile.tlv
-rw-r--r--  1 certbase ccmbase 4930 Jun 30 15:38
/usr/local/cm/tftp/ITLFile.tlv

Permissions issue.  On Matthew's box:
[root at server ~]# ps -eaf | grep -iE "TFTP|TVS"
certbase 31007 17132  0 15:35 ?        00:00:00 /usr/local/cm/bin/tvs
ctftp    31574 17132  0 15:36 ?        00:00:02 /usr/local/cm/bin/ctftp

Looks like ctftp service most likely wrote or created ITL file on May 25.
The owner of ITLFile.tlv should be certbase not ctftp.  We did:
1. chown certbase ITLFile.tlv - this set the correct owner
2. restarted TVS service - TVS regenerated ITL file and successfully wrote
it to the file system
3. restart TFTP service so it could pickup the new ITL file from the file
system.

After this phones successfully registered. So far there is at least one new
bug out of this:
CSCtr27100 TVS inaccurately reports New ITL File has been generated

In CUCM 8.6 ctftp does indeed generate the ITL file.  In 8.5.1.11900-21 (aka
8.5.1su1) that Matthew is running ITL file should be generated by TVS.

Regards,
Wes

On 6/30/2011 4:17 PM, Matthew Loraditch wrote: 

Well I just finished what amounted to 5 hours on the phone and 6 hours for
Cisco. Most of that with Wes and apparently Ryan Ratliff and Jason Burns
stopped by for a while as well!

Anyway we got it fixed. Wes said he is going to do a write up but the gist
was the ITL cert was written by the wrong service and thus not matching up
and they had to go in with root and do something so that right service could
properly create it.

Major Kudos to Wes and if we are ever within 50 miles or less of each other
drinks are in order!

Matthew Loraditch, CCVP, CCNA, CCDA
1965 Greenspring Drive

Timonium, MD 21093 
 <mailto:support at heliontechnologies.com> support at heliontechnologies.com
(p)  <tel:%28410%29%20252-8830> (410) 252-8830
(F)  <tel:%28443%29%20541-1593> (443) 541-1593

Visit us at  <http://www.heliontechnologies.com/> www.heliontechnologies.com

Support Issue? Email  <mailto:support at heliontechnologies.com>
support at heliontechnologies.com for fast assistance!

From:  <mailto:cisco-voip-bounces at puck.nether.net>
cisco-voip-bounces at puck.nether.net [
<mailto:cisco-voip-bounces at puck.nether.net>
mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Matthew Loraditch
Sent: Thursday, June 30, 2011 11:24 AM
To: Lelio Fulgenzi
Cc:  <mailto:cisco-voip at puck.nether.net> cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS, etc

Wes is helping out and watching my case so far we have just verified normal
settings, regenerated the tomcat certs to no avail and then totally factory
defaulted the phone (the 123456789*0#) and now the phone comes up
unprovisioned and won't connect at all!

Engineers seem pretty stumped

Matthew Loraditch, CCVP, CCNA, CCDA
1965 Greenspring Drive

Timonium, MD 21093 
 <mailto:support at heliontechnologies.com> support at heliontechnologies.com
(p)  <tel:%28410%29%20252-8830> (410) 252-8830
(F)  <tel:%28443%29%20541-1593> (443) 541-1593

Visit us at  <http://www.heliontechnologies.com/> www.heliontechnologies.com

Support Issue? Email  <mailto:support at heliontechnologies.com>
support at heliontechnologies.com for fast assistance!

From: Lelio Fulgenzi [ <mailto:lelio at uoguelph.ca> mailto:lelio at uoguelph.ca] 
Sent: Thursday, June 30, 2011 11:21 AM
To: Matthew Loraditch
Cc: Matthew Loraditch;  <mailto:cisco-voip at puck.nether.net>
cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS, etc

Let us know how that goes...

Sent from my iPhone

On Jun 30, 2011, at 9:09 AM, Matthew Loraditch
<MLoraditch at heliontechnologies.com> wrote:

Well I have done what Lelio suggested, I have rebooted the cluster. I
noticed from the phone logs and in security settings that the certs they are
getting say the hostname so I changed the CCMs from IP back to hostname.
Still no dice.

Time to open a TAC case!

Matthew Loraditch, CCVP, CCNA, CCDA
1965 Greenspring Drive

Timonium, MD 21093 
 <mailto:support at heliontechnologies.com> support at heliontechnologies.com
(p)  <tel:%28410%29%20252-8830> (410) 252-8830
(F)  <tel:%28443%29%20541-1593> (443) 541-1593

Visit us at  <http://www.heliontechnologies.com/> www.heliontechnologies.com

Support Issue? Email  <mailto:support at heliontechnologies.com>
support at heliontechnologies.com for fast assistance!

From:  <mailto:cisco-voip-bounces at puck.nether.net>
cisco-voip-bounces at puck.nether.net [
<mailto:cisco-voip-bounces at puck.nether.net>
mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Matthew Loraditch
Sent: Wednesday, June 29, 2011 9:57 PM
To: Lelio Fulgenzi
Cc:  <mailto:cisco-voip at puck.nether.net> cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS, etc

Will give it a whirl in the am

Matthew Loraditch, CCVP, CCNA, CCDA
1965 Greenspring Drive

Timonium, MD 21093 
 <mailto:support at heliontechnologies.com> support at heliontechnologies.com
(p)  <tel:%28410%29%20252-8830> (410) 252-8830
(F)  <tel:%28443%29%20541-1593> (443) 541-1593

Visit us at  <http://www.heliontechnologies.com/> www.heliontechnologies.com

Support Issue? Email  <mailto:support at heliontechnologies.com>
support at heliontechnologies.com for fast assistance!

From: Lelio Fulgenzi [ <mailto:lelio at uoguelph.ca> mailto:lelio at uoguelph.ca] 
Sent: Wednesday, June 29, 2011 9:43 PM
To: Matthew Loraditch
Cc:  <mailto:wsisk at cisco.com> wsisk at cisco.com;
<mailto:cisco-voip at puck.nether.net> cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS, etc

Try the service parameter I mentioned. 

If you do a search on the archives, Wes posted a link to the doc bug.  

Sent from my iPhone

On Jun 29, 2011, at 9:31 PM, Matthew Loraditch
<MLoraditch at heliontechnologies.com> wrote:

7942s and 7945s so far, in re someone else's email have restarted tftp etc
already.

Matthew Loraditch, CCVP, CCNA, CCDA
1965 Greenspring Drive

Timonium, MD 21093 
 <mailto:support at heliontechnologies.com> support at heliontechnologies.com
(p)  <tel:%28410%29%20252-8830> (410) 252-8830
(F)  <tel:%28443%29%20541-1593> (443) 541-1593

Visit us at  <http://www.heliontechnologies.com/> www.heliontechnologies.com

Support Issue? Email  <mailto:support at heliontechnologies.com>
support at heliontechnologies.com for fast assistance!

From: Lelio Fulgenzi  <mailto:[mailto:lelio at uoguelph.ca]>
[mailto:lelio at uoguelph.ca] 
Sent: Wednesday, June 29, 2011 7:50 PM
To: Matthew Loraditch
Cc:  <mailto:wsisk at cisco.com> wsisk at cisco.com;
<mailto:cisco-voip at puck.nether.net> cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS, etc

Are these 7941/61? I had a problem where I had to change the service
provisioning service parameter to external to make this work. 

Sent from my iPhone

On Jun 29, 2011, at 6:23 PM, Matthew Loraditch
<MLoraditch at heliontechnologies.com> wrote:

Googled and found that and it's not working..

Matthew Loraditch, CCVP, CCNA, CCDA
1965 Greenspring Drive

Timonium, MD 21093

support at heliontechnologies.com
(p) (410) 252-8830 <tel:%28410%29%20252-8830> 
(F) (443) 541-1593 <tel:%28443%29%20541-1593> 

Visit us at www.heliontechnologies.com

Support Issue? Email support at heliontechnologies.com
for fast assistance!

From: Wes Sisk
[mailto:wsisk at cisco.com] 
Sent: Wednesday, June 29, 2011 5:24 PM
To: Matthew Loraditch
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] Phones Not Getting Auth, Idle, Services URLS, etc

These lines pretty much say it all. verification of the downloaded file
failed. Delete the CTL/ITL from the phone and try again.
https://supportforums.cisco.com/docs/DOC-15799#Manual_ITL_Delete

Regards,
Wes

On 6/29/2011 5:03 PM, Matthew Loraditch wrote: 

1715: ERR 16:59:35.170584 SECD: EROR:verifyFile: sgn verify file failed
</usr/ram/SEP00260BD749E9.cnf.xml>, errclass 8, errcode 19 (signer
not in CTL)

1716: ERR 16:59:35.171327 SECD: EROR:verifyFile: verify FAILED,
</usr/ram/SEP00260BD749E9.cnf.xml>

Sent from my Android phone using TouchDown (www.nitrodesk.com) 

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

************************************
This email may contain privileged and/or confidential information that is
intended solely for the use of the addressee.  If you are not the intended
recipient or entity, you are strictly prohibited from disclosing, copying,
distributing or using any of the information contained in the transmission.
If you received this communication in error, please contact the sender
immediately and destroy the material in its entirety, whether electronic or
hard copy.  This communication may contain nonpublic personal information
about consumers subject to the restrictions of the Gramm-Leach-Bliley Act
and the Sarbanes-Oxley Act.  You may not directly or indirectly reuse or
disclose such information for any purpose other than to provide the services
for which you are receiving the information.
There are risks associated with the use of electronic transmission.  The
sender of this information does not control the method of transmittal or
service providers and assumes no duty or obligation for the security,
receipt, or third party interception of this transmission.
************************************

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20110816/d8af70e2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2768 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20110816/d8af70e2/attachment.jpg>