[cisco-voip] 9.3.1 , TVS, and IPv6

Mon Nov 5 11:46:49 EST 2012

Absolutely correct Dennis ;)

That is a valid approach as long as the phone 'trusts' the callmanager.pem certificate used by the TFTP service, however if it does not you need to get the phones ITL file updated before you can remove it. There are two main ways to do that:

1) Get the TVS service to update the phones ITL File (sometimes a restart of the TVS Service, or regenerating the Callmanager.pem cert works)
2) Delete the ITL file on the phone

Option 1 doesn't always work and can cause further issues that need to be considered/managed (Catch-22)
Option 2 generally always works

It's too easy to get stuck in a catch-22 with SBD related issues, the combination of certificates, phone configuration and TVS service all lead back to each other in a nasty loop, from our experience the easiest way to break the look is delete the ITL file on the phone's user interface.

I'm not stating that remotely deleting the ITL file from all your phones is the ideal solution to the problem, just that in reality it is the best solution that is proven to work when other approaches can fail at points.

Thanks

Stephen

On 5 Nov 2012, at 16:30, "Heim, Dennis" <Dennis.Heim at wwt.com<mailto:Dennis.Heim at wwt.com>>
 wrote:

The article also talks about setting the rollback parameter to true, which will erase the ITL on the phones. Obviously this requires that your phones can still register.

Dennis Heim | Sr. UC Engineer
World Wide Technology | 314.212.1814 | dennis.heim at wwt.com<mailto:dennis.heim at wwt.com>
“Creating Impact, Ignition & Scalability”

From: cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net> [mailto:cisco-voip-bounces at puck.nether.net<mailto:voip-bounces at puck.nether.net>] On Behalf Of Stephen Welsh
Sent: Monday, November 05, 2012 11:06 AM
To: Adam Pawlowski
Cc: <cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>>
Subject: Re: [cisco-voip] 9.3.1 , TVS, and IPv6

Hi Adam,

As you stated (very nicely :), everyone that upgrades to (or between) UCM 8 & 9 versions needs to understand SBD. Unfortunately even if you do everything right you can still get hit by a SBD/ITL problems, there are several different ways to get caught out even if you go 100% by the book.
In our experience the best way to handle almost any SBD issue is deleting the ITL File, there are some steps to prevent issues, but they are never 100% fool-proof, the best thing to do is be prepared to 'easily' manage/delete those pesky ITL files ;)

If you haven't already found this, the best information on Security by Default is by Jason Burns:

https://supportforums.cisco.com/docs/DOC-17679

I completely appreciate your point of view that the last thing you should "have" to do is delete the ITL file, if you read Jason's document this will give you the best understanding and chance to eliminate (or minimise) this likely hood. However I always recommend that you have a Plan-B, Unified FX has devised an approach (a way to configure your cluster) that will ensure that you will never need to physically go to a phone to delete an ITL files even in the worst and most rare SDB Issues. Believe me the issue you have at the moment is nothing compared to some of the SBD problems we have had to solve, at least your phones are still registered ;)

Thanks

Stephen Welsh, CCIE #12345
CTO
Unified FX
http://www.unifiedfx.com

On 5 Nov 2012, at 15:37, Adam Pawlowski <ajp26 at buffalo.edu<mailto:ajp26 at buffalo.edu>>
 wrote:

Stephen,

     Removing the ITL file from the phone does seem to allow it to grab a new one, since it doesn’t need to verify it. When the phone is booting up, it lists, from a file on its flash, that it has no TVS servers, until it has read the configuration file and established the CM nodes from that device pool. TVS is running on all hosts in the cluster, including the TFTP, which it seems to want to fall back to, to verify its configuration and ITL initially. For whatever reason it insists on using the TFTP, yet, it wants to connect via IPv6 which just can’t work and doesn’t. It’s placed me into a situation where I want to turn off V6 but, have to do so in the phone’s config, and can’t change the phone’s config until I turn off V6 (or erase the ITL).

     I will inspect the ITL on the phone again to see what’s going on – the nodes should all be listed but the ITLs hash has changed. I’d love to understand just what is going on with this. As we were talking earlier about here, the license documentation for the 8 upgrades should have come with a singing telegram or at least a stack of bright red paper drawing our attention to this feature. There’s a lot that’s new in the UCM and it seems easy to get underwater on the everything that comes up, but this just seems too easy to run afoul of bugs or problems which can hose over your cluster. If we have to erase, we have to erase, but, I don’t want to get into the habit of planning to or having to erase security certificates if there’s a way to correct the issue.

Thanks much though for your time and reply

Adam

From: Stephen Welsh [mailto:stephen.welsh at unifiedfx.com<http://unifiedfx.com>]
Sent: Monday, November 05, 2012 9:09 AM
To: Pawlowski, Adam
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] 9.3.1 , TVS, and IPv6

Hi Adam,

Are you sure the ipv6 thing is not a red herring?

Have you tried removing the ITL File from the phone and/or restarting the TVS service?

If you are not aware the choice of TVS service (node IP Address) is based on the Call Manager Group configured on the phone, you could try adding all nodes to the phones CM Group to see if that helps the phone to contact a TVS service it has a matching ITL entry for.

If you are still stuck I'm happy to host a WebEx session to share my SDB & ITL experiences, I'm the original author of PhoneView (http://www.unifiedfx.com), it's been used to help a LOT of people with SBD related issues, never-mind countless UCM installations and upgrades.

In-case you do need to delete/manage your ITL Files, or even just get a proper view/handle on the phone firmware version of your estate you should give PhoneView a try:

http://www.unifiedfx.com/phoneview/trial

Thanks

Stephen

On 5 Nov 2012, at 13:51, "Pawlowski, Adam" <ajp26 at buffalo.edu<mailto:ajp26 at buffalo.edu>>
 wrote:

Morning list,

     From what I can read, TVS has been a thrill ride for those of us lucky enough to run afoul of SBD. I’m hoping we haven’t run into such a situation here but I have a question. We just pushed 9.3.1 out to devices from 9.1.1SR1, with peer firmware sharing enabled. This went miserably and left a lot of devices stranded at 9.1.1 or at an “Upgrading” screen. Working with TAC on that but so far nothing. We began receiving reports from end users that their directories were missing, they can’t change ringers, etc. Last time this happened we had phones that had picked up a bum ITL from a partial rollout, and we had to erase them by hand.

     This time that shouldn’t have been the case – it was just a firmware upgrade. However, it looks like some of the phones that have gone to 9.3.1 have decided that they want to use IPv6 when talking to the TFTP to verify their initial configurations, when they have no TVS server list built on the device. In looking at the phone console logs, you can see that the device is in “IP mode 1” and tries to connect to TFTP, say “192.168.0.1 :: “ which is obviously not a V6 address. We’re not running V6 so it has no address bound. No traffic leaves the phone, but, the phone says it timed out (EAGAIN) connecting to the TFTP and won’t verify the ITL/CFG/etc.

      What I’m looking at it is setting V6 to off at the cluster, but, I don’t see any way to repair this on affected devices (could be a large amount) other than erasing the ITL, or deploying IPv6. Given the leg work that could go into identifying and taking care of these devices, it’s arguable as to which one would be harder at this point.

      Any comment on this with this firmware? Anyone else run into this miserable trouble?

Adam Pawlowski
SUNYAB Network and Classroom Services
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20121105/db688de1/attachment.html>