[cisco-voip] 9.3.1 , TVS, and IPv6
Jason Burns
burns.jason at gmail.com
Wed Nov 7 13:38:43 EST 2012
Adam,
I've seen this exact same thing with some of my customers. Here's what I
think caused the whole IPv6 problem on the phone:
1. CUCM cluster was upgraded.
2. Pub and TFTP servers were rebooted.
3. Backup subs were rebooted.
4. Primary subs were rebooted.
5. Phones reset and try to get a TFTP file from the TFTP server.
Here's where the problem is. The phones tried to get a TFTP file, but the
TFTP server hadn't completed database replication yet. The phones
downloaded a non complete config file, or a non complete ITL file. This ITL
or config file (i never did figure out which) put the phone into a mode
where for every single TVS request it tried to use the IPv6 stack on the
phone. You'd see on the phone console logs that the phone tried to connect
to TVS and it timed out. You'd never see any packets leave the phone to TCP
2445 in a packet capture.
I believe we fixed the problem by shutting down the primary TFTP server and
letting the phones use the backup TFTP server. This worked because the
phones had an ITL that was signed by the backup TFTP server so they trusted
the backup TFTP server's key. The ITL file they had though, or the config
file wouldn't let TVS work.. we needed to find the server that signed the
phone's file and get the phone to go BACK to that original server so it
could get a complete config file.
Either do that or erase the ITL manually from the phone (or use an
automated tool). I'm including Tom here because he worked on this problem
and may remember the details more accurately. I don't know if we ever found
a bug for this.
There are automated tools that make erasing the ITL easier, but for your
case if you have more than one TFTP server try either:
1. Shutting down the primary TFTP service and resetting the phones so they
go to backup.
2. Changing the DHCP scope so they go to the backup.
3. Change the TFTP address manually in a phone to point to another server.
Hope that points you in the right direction.
-Jason
On Mon, Nov 5, 2012 at 11:46 AM, Stephen Welsh
<stephen.welsh at unifiedfx.com>wrote:
> Absolutely correct Dennis ;)
>
> That is a valid approach as long as the phone 'trusts' the
> callmanager.pem certificate used by the TFTP service, however if it does
> not you need to get the phones ITL file updated before you can remove it.
> There are two main ways to do that:
>
> 1) Get the TVS service to update the phones ITL File (sometimes a
> restart of the TVS Service, or regenerating the Callmanager.pem cert works)
> 2) Delete the ITL file on the phone
>
> Option 1 doesn't always work and can cause further issues that need to
> be considered/managed (Catch-22)
> Option 2 generally always works
>
> It's too easy to get stuck in a catch-22 with SBD related issues, the
> combination of certificates, phone configuration and TVS service all lead
> back to each other in a nasty loop, from our experience the easiest way to
> break the look is delete the ITL file on the phone's user interface.
>
> I'm not stating that remotely deleting the ITL file from all your phones
> is the ideal solution to the problem, just that in reality it is the best
> solution that is proven to work when other approaches can fail at points.
>
> Thanks
>
> Stephen
>
> On 5 Nov 2012, at 16:30, "Heim, Dennis" <Dennis.Heim at wwt.com>
> wrote:
>
> The article also talks about setting the rollback parameter to true,
> which will erase the ITL on the phones. Obviously this requires that your
> phones can still register.****
>
> *Dennis Heim | Sr. UC Engineer***
> World Wide Technology | 314.212.1814 | dennis.heim at wwt.com**
> “Creating Impact, Ignition & Scalability”****
>
> *From:* cisco-voip-bounces at puck.nether.net [mailto:cisco-
> voip-bounces at puck.nether.net] *On Behalf Of *Stephen Welsh
> *Sent:* Monday, November 05, 2012 11:06 AM
> *To:* Adam Pawlowski
> *Cc:* <cisco-voip at puck.nether.net>
> *Subject:* Re: [cisco-voip] 9.3.1 , TVS, and IPv6****
> ** **
> Hi Adam,****
> ** **
> As you stated (very nicely :), everyone that upgrades to (or between)
> UCM 8 & 9 versions needs to understand SBD. Unfortunately even if you do
> everything right you can still get hit by a SBD/ITL problems, there are
> several different ways to get caught out even if you go 100% by the book.*
> ***
> In our experience the best way to handle almost any SBD issue is
> deleting the ITL File, there are some steps to prevent issues, but they are
> never 100% fool-proof, the best thing to do is be prepared to 'easily'
> manage/delete those pesky ITL files ;)****
> ** **
> If you haven't already found this, the best information on Security by
> Default is by Jason Burns:****
> ** **
> https://supportforums.cisco.com/docs/DOC-17679****
> ** **
> I completely appreciate your point of view that the last thing you
> should "have" to do is delete the ITL file, if you read Jason's document
> this will give you the best understanding and chance to eliminate (or
> minimise) this likely hood. However I always recommend that you have a
> Plan-B, Unified FX has devised an approach (a way to configure your
> cluster) that will ensure that you will never need to physically go to a
> phone to delete an ITL files even in the worst and most rare SDB Issues.
> Believe me the issue you have at the moment is nothing compared to some of
> the SBD problems we have had to solve, at least your phones are still
> registered ;)****
> ** **
> Thanks****
> ** **
> Stephen Welsh, CCIE #12345****
> CTO****
> Unified FX****
> http://www.unifiedfx.com****
> ** **
> On 5 Nov 2012, at 15:37, Adam Pawlowski <ajp26 at buffalo.edu>****
> wrote:****
>
>
> ****
> Stephen,****
> ****
> Removing the ITL file from the phone does seem to allow it to grab
> a new one, since it doesn’t need to verify it. When the phone is booting
> up, it lists, from a file on its flash, that it has no TVS servers, until
> it has read the configuration file and established the CM nodes from that
> device pool. TVS is running on all hosts in the cluster, including the
> TFTP, which it seems to want to fall back to, to verify its configuration
> and ITL initially. For whatever reason it insists on using the TFTP, yet,
> it wants to connect via IPv6 which just can’t work and doesn’t. It’s placed
> me into a situation where I want to turn off V6 but, have to do so in the
> phone’s config, and can’t change the phone’s config until I turn off V6 (or
> erase the ITL).****
> ****
> I will inspect the ITL on the phone again to see what’s going on –
> the nodes should all be listed but the ITLs hash has changed. I’d love to
> understand just what is going on with this. As we were talking earlier
> about here, the license documentation for the 8 upgrades should have come
> with a singing telegram or at least a stack of bright red paper drawing our
> attention to this feature. There’s a lot that’s new in the UCM and it seems
> easy to get underwater on the everything that comes up, but this just seems
> too easy to run afoul of bugs or problems which can hose over your cluster.
> If we have to erase, we have to erase, but, I don’t want to get into the
> habit of planning to or having to erase security certificates if there’s a
> way to correct the issue.****
> ****
> Thanks much though for your time and reply****
> ****
> Adam****
> ****
> *From:* Stephen Welsh [mailto:stephen.welsh at unifiedfx.com]
> *Sent:* Monday, November 05, 2012 9:09 AM
> *To:* Pawlowski, Adam
> *Cc:* cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] 9.3.1 , TVS, and IPv6****
> ****
> Hi Adam,****
> ****
> Are you sure the ipv6 thing is not a red herring?****
> ****
> Have you tried removing the ITL File from the phone and/or restarting
> the TVS service?****
> ****
> If you are not aware the choice of TVS service (node IP Address) is
> based on the Call Manager Group configured on the phone, you could try
> adding all nodes to the phones CM Group to see if that helps the phone to
> contact a TVS service it has a matching ITL entry for.****
> ****
> If you are still stuck I'm happy to host a WebEx session to share my
> SDB & ITL experiences, I'm the original author of PhoneView (
> http://www.unifiedfx.com), it's been used to help a LOT of people with
> SBD related issues, never-mind countless UCM installations and upgrades.**
> **
> ****
> In-case you do need to delete/manage your ITL Files, or even just get a
> proper view/handle on the phone firmware version of your estate you should
> give PhoneView a try:****
> ****
> http://www.unifiedfx.com/phoneview/trial****
> ****
> Thanks****
> ****
> Stephen****
> ****
> On 5 Nov 2012, at 13:51, "Pawlowski, Adam" <ajp26 at buffalo.edu>****
> wrote:****
>
>
>
> ****
> Morning list,****
> ****
> From what I can read, TVS has been a thrill ride for those of us
> lucky enough to run afoul of SBD. I’m hoping we haven’t run into such a
> situation here but I have a question. We just pushed 9.3.1 out to devices
> from 9.1.1SR1, with peer firmware sharing enabled. This went miserably and
> left a lot of devices stranded at 9.1.1 or at an “Upgrading” screen.
> Working with TAC on that but so far nothing. We began receiving reports
> from end users that their directories were missing, they can’t change
> ringers, etc. Last time this happened we had phones that had picked up a
> bum ITL from a partial rollout, and we had to erase them by hand.****
> ****
> This time that shouldn’t have been the case – it was just a
> firmware upgrade. However, it looks like some of the phones that have gone
> to 9.3.1 have decided that they want to use IPv6 when talking to the TFTP
> to verify their initial configurations, when they have no TVS server list
> built on the device. In looking at the phone console logs, you can see that
> the device is in “IP mode 1” and tries to connect to TFTP, say “192.168.0.1
> :: “ which is obviously not a V6 address. We’re not running V6 so it has no
> address bound. No traffic leaves the phone, but, the phone says it timed
> out (EAGAIN) connecting to the TFTP and won’t verify the ITL/CFG/etc.****
> ****
> What I’m looking at it is setting V6 to off at the cluster, but,
> I don’t see any way to repair this on affected devices (could be a large
> amount) other than erasing the ITL, or deploying IPv6. Given the leg work
> that could go into identifying and taking care of these devices, it’s
> arguable as to which one would be harder at this point.****
> ****
> Any comment on this with this firmware? Anyone else run into this
> miserable trouble?****
> ****
> Adam Pawlowski****
> SUNYAB Network and Classroom Services****
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip****
>
>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20121107/5d8658aa/attachment.html>
More information about the cisco-voip
mailing list