[cisco-voip] 9.3.1 , TVS, and IPv6

[Adam Pawlowski]

This does seem to be a way around this trouble. 

>From the set that wasn't working correctly, from the CLI, we were able to
get this output:

001B2A20FFFF> show tvs
Priority : 0
TVS IPv4 Address : 1.2.3.4
TVS IPv6 Address :
TVS Port : 2445
Priority : 1
TVS IPv4 Address : 2.3.4.5
TVS IPv6 Address :
TVS Port : 2445

>From a working set though:

001B2A20FFFF> show tvs
Priority : 0
TVS IPv4 Address : 1.2.3.4
TVS IPv6 Address :
TVS Port : 2445
IP Mode : 0
IP Pref : 0
DSCP value : 96
Priority : 1
TVS IPv4 Address : 2.3.4.5
TVS IPv6 Address :
TVS Port : 2445
IP Mode : 0
IP Pref : 0
DSCP value : 96

Since we moved from 9.1.1 to 9.3.1, I wonder if the format of this file
changed, and wasn't successfully updated on some phones for whatever reason.
One the phone was "repaired" the file reflects the latter format. I wonder
if it's that this file is not correctly formatted that causes it causes that
fallback to happen, since I could see some errors about reading the tvs conf
file in the phone's log. Then again, the file may not be formatted correctly
because of some other thing, either way, I'm not sure I'm going to find a
cause. This is life on the bleeding edge of things, I guess, running into
anomalies. The TFTP "fix" works, and makes sense. TAC was going down the bad
ITL route, which I would guess is the basket most people are in with ITL
problems, but, it wasn't that at all. The ITL would have successfully
updated if it bothered to contact the host. Wireshark showed the phone not
emitting any traffic at all, (I didn't try configuring the phone with an
IPv6 address itself to see if it started spouting out traffic but there was
no V6 address for the TVS/TFTP present at that time), so it had no chance to
upgrade.

I'm not convinced this has anything to do with the UCM and not the software
on the phone, but in this case this investigation was being tied to the UCM
version, which we need to upgrade, I guess, to pursue this further. If it
comes up again I'll open a case again to see what they say.

Anyways, we'll probably still end up picking up some software to deal with
the ITLs so we are not in panic mode if we do run into problems with it
later on.

Thanks all for your help

Adam

> -----Original Message-----
> From: Tom Piscitell (tpiscite) [mailto:tpiscite at cisco.com]
> Sent: Wednesday, November 07, 2012 4:22 PM
> To: Jason Burns
> Cc: Stephen Welsh; Tom Piscitell (tpiscite); Heim, Dennis; <cisco-
> voip at puck.nether.net>; Adam Pawlowski
> Subject: Re: [cisco-voip] 9.3.1 , TVS, and IPv6
> 
> Hey Adam,
> 
> This does sounds like the situation me and Jason have ran into a couple
> times. Unfortunately we were never able to reproduce it so we never
> understood exactly what happened to put the phones in this situation.
> 
> However, more specific to your situation, a phone will only need to use
TVS
> to verify its configuration file if it is using a TFTP server that it did
not get its
> ITL file from. In the ITL file there is only one CallManager.pem (the
certificate
> that signs the config files). So if you have an ITL from Server A, and
then try
> and use Server B for TFTP, you will need to use TVS. However, as long as
you
> use Server A for TFTP, there will be no need for TVS.
> 
> So with that said, it sounds like the phones have an ITL file from some
server
> other than their primary TFTP server (192.168.0.1??). Alternatively, they
may
> have an older version of the ITL from the primary TFTP server because the
> CallManager.pem was recently regenerated. Let's assume its the former
> since the only change was a firmware upgrade. To find out what server the
> ITL file came from, we need to compare the hash that shows up in the phone
> UI (Settings > Security Settings > Trust List > ITL). You will then need
to
> compare that to the ITL file on the servers. Here is a quick one-liner to
get
> the hash of the current ITL on CUCM:
> 
> tpiscite-mac:CUCMFolder tompiscitell$ curl
> http://14.48.30.101:6970/ITLFile.tlv 2>/dev/null | md5
> 48bf091242356d56dcdbb4aca5d9095e
> 
> Note: If you want to do the same for the CTL just replace "ITLFile.tlv"
with
> "CTLFile.tlv".
> 
> Once you find the TFTP server that the ITL file came from, you just need
to
> point the phones at that (Alternate TFTP, DHCP Option 150, shutting down
> the other TFTP server, etc), and have it register. If you cannot find a
matching
> ITL file on any of the servers, then the phones probably have an old ITL
file
> and SBD was "broken" for quite a while, but the firmware upgrade
> uncovered it.
> 
> Hope this helps.
> 
> Tom Piscitell
> Customer Support Engineer
> Unified Communications Infrastructure
> +1-919-574-0081, 9am-5pm EST (GMT -5)
> 
> On Nov 7, 2012, at 1:38 PM, Jason Burns <burns.jason at gmail.com> wrote:
> 
> > Adam,
> >
> > I've seen this exact same thing with some of my customers. Here's what I
> think caused the whole IPv6 problem on the phone:
> >
> > 1. CUCM cluster was upgraded.
> > 2. Pub and TFTP servers were rebooted.
> > 3. Backup subs were rebooted.
> > 4. Primary subs were rebooted.
> > 5. Phones reset and try to get a TFTP file from the TFTP server.
> >
> > Here's where the problem is. The phones tried to get a TFTP file, but
the
> TFTP server hadn't completed database replication yet. The phones
> downloaded a non complete config file, or a non complete ITL file. This
ITL or
> config file (i never did figure out which) put the phone into a mode where
for
> every single TVS request it tried to use the IPv6 stack on the phone.
You'd
> see on the phone console logs that the phone tried to connect to TVS and
it
> timed out. You'd never see any packets leave the phone to TCP 2445 in a
> packet capture.
> >
> > I believe we fixed the problem by shutting down the primary TFTP server
> and letting the phones use the backup TFTP server. This worked because the
> phones had an ITL that was signed by the backup TFTP server so they
trusted
> the backup TFTP server's key. The ITL file they had though, or the config
file
> wouldn't let TVS work.. we needed to find the server that signed the
phone's
> file and get the phone to go BACK to that original server so it could get
a
> complete config file.
> >
> > Either do that or erase the ITL manually from the phone (or use an
> automated tool). I'm including Tom here because he worked on this problem
> and may remember the details more accurately. I don't know if we ever
> found a bug for this.
> >
> > There are automated tools that make erasing the ITL easier, but for your
> case if you have more than one TFTP server try either:
> >
> > 1. Shutting down the primary TFTP service and resetting the phones so
> they go to backup.
> > 2. Changing the DHCP scope so they go to the backup.
> > 3. Change the TFTP address manually in a phone to point to another
server.
> >
> > Hope that points you in the right direction.
> >
> > -Jason
> >
> >
> >
> > On Mon, Nov 5, 2012 at 11:46 AM, Stephen Welsh
> <stephen.welsh at unifiedfx.com> wrote:
> > Absolutely correct Dennis ;)
> >
> > That is a valid approach as long as the phone 'trusts' the
callmanager.pem
> certificate used by the TFTP service, however if it does not you need to
get
> the phones ITL file updated before you can remove it. There are two main
> ways to do that:
> >
> > 1) Get the TVS service to update the phones ITL File (sometimes a
> > restart of the TVS Service, or regenerating the Callmanager.pem cert
> > works)
> > 2) Delete the ITL file on the phone
> >
> > Option 1 doesn't always work and can cause further issues that need to
> > be considered/managed (Catch-22) Option 2 generally always works
> >
> > It's too easy to get stuck in a catch-22 with SBD related issues, the
> combination of certificates, phone configuration and TVS service all lead
back
> to each other in a nasty loop, from our experience the easiest way to
break
> the look is delete the ITL file on the phone's user interface.
> >
> > I'm not stating that remotely deleting the ITL file from all your phones
is the
> ideal solution to the problem, just that in reality it is the best
solution that is
> proven to work when other approaches can fail at points.
> >
> > Thanks
> >
> > Stephen
> >
> > On 5 Nov 2012, at 16:30, "Heim, Dennis" <Dennis.Heim at wwt.com>
> >  wrote:
> >
> >> The article also talks about setting the rollback parameter to true,
which
> will erase the ITL on the phones. Obviously this requires that your phones
> can still register.
> >>
> >> Dennis Heim | Sr. UC Engineer
> >> World Wide Technology | 314.212.1814 | dennis.heim at wwt.com
> "Creating
> >> Impact, Ignition & Scalability"
> >>
> >> From: cisco-voip-bounces at puck.nether.net
> >> [mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Stephen
> >> Welsh
> >> Sent: Monday, November 05, 2012 11:06 AM
> >> To: Adam Pawlowski
> >> Cc: <cisco-voip at puck.nether.net>
> >> Subject: Re: [cisco-voip] 9.3.1 , TVS, and IPv6
> >>
> >> Hi Adam,
> >>
> >> As you stated (very nicely :), everyone that upgrades to (or between)
> UCM 8 & 9 versions needs to understand SBD. Unfortunately even if you do
> everything right you can still get hit by a SBD/ITL problems, there are
several
> different ways to get caught out even if you go 100% by the book.
> >> In our experience the best way to handle almost any SBD issue is
> >> deleting the ITL File, there are some steps to prevent issues, but
> >> they are never 100% fool-proof, the best thing to do is be prepared
> >> to 'easily' manage/delete those pesky ITL files ;)
> >>
> >> If you haven't already found this, the best information on Security by
> Default is by Jason Burns:
> >>
> >> https://supportforums.cisco.com/docs/DOC-17679
> >>
> >> I completely appreciate your point of view that the last thing you
> >> should "have" to do is delete the ITL file, if you read Jason's
> >> document this will give you the best understanding and chance to
> >> eliminate (or minimise) this likely hood. However I always recommend
> >> that you have a Plan-B, Unified FX has devised an approach (a way to
> >> configure your cluster) that will ensure that you will never need to
> >> physically go to a phone to delete an ITL files even in the worst and
> >> most rare SDB Issues. Believe me the issue you have at the moment is
> >> nothing compared to some of the SBD problems we have had to solve, at
> >> least your phones are still registered ;)
> >>
> >> Thanks
> >>
> >> Stephen Welsh, CCIE #12345
> >> CTO
> >> Unified FX
> >> http://www.unifiedfx.com
> >>
> >> On 5 Nov 2012, at 15:37, Adam Pawlowski <ajp26 at buffalo.edu>
> >>  wrote:
> >>
> >>
> >> Stephen,
> >>
> >>      Removing the ITL file from the phone does seem to allow it to grab
a
> new one, since it doesn't need to verify it. When the phone is booting up,
it
> lists, from a file on its flash, that it has no TVS servers, until it has
read the
> configuration file and established the CM nodes from that device pool. TVS
is
> running on all hosts in the cluster, including the TFTP, which it seems to
want
> to fall back to, to verify its configuration and ITL initially. For
whatever reason
> it insists on using the TFTP, yet, it wants to connect via IPv6 which just
can't
> work and doesn't. It's placed me into a situation where I want to turn off
V6
> but, have to do so in the phone's config, and can't change the phone's
config
> until I turn off V6 (or erase the ITL).
> >>
> >>      I will inspect the ITL on the phone again to see what's going on -
the
> nodes should all be listed but the ITLs hash has changed. I'd love to
> understand just what is going on with this. As we were talking earlier
about
> here, the license documentation for the 8 upgrades should have come with a
> singing telegram or at least a stack of bright red paper drawing our
attention
> to this feature. There's a lot that's new in the UCM and it seems easy to
get
> underwater on the everything that comes up, but this just seems too easy
to
> run afoul of bugs or problems which can hose over your cluster. If we have
to
> erase, we have to erase, but, I don't want to get into the habit of
planning to
> or having to erase security certificates if there's a way to correct the
issue.
> >>
> >> Thanks much though for your time and reply
> >>
> >> Adam
> >>
> >> From: Stephen Welsh [mailto:stephen.welsh at unifiedfx.com]
> >> Sent: Monday, November 05, 2012 9:09 AM
> >> To: Pawlowski, Adam
> >> Cc: cisco-voip at puck.nether.net
> >> Subject: Re: [cisco-voip] 9.3.1 , TVS, and IPv6
> >>
> >> Hi Adam,
> >>
> >> Are you sure the ipv6 thing is not a red herring?
> >>
> >> Have you tried removing the ITL File from the phone and/or restarting
the
> TVS service?
> >>
> >> If you are not aware the choice of TVS service (node IP Address) is
based
> on the Call Manager Group configured on the phone, you could try adding
all
> nodes to the phones CM Group to see if that helps the phone to contact a
> TVS service it has a matching ITL entry for.
> >>
> >> If you are still stuck I'm happy to host a WebEx session to share my
SDB &
> ITL experiences, I'm the original author of PhoneView
> (http://www.unifiedfx.com), it's been used to help a LOT of people with
SBD
> related issues, never-mind countless UCM installations and upgrades.
> >>
> >> In-case you do need to delete/manage your ITL Files, or even just get a
> proper view/handle on the phone firmware version of your estate you
> should give PhoneView a try:
> >>
> >> http://www.unifiedfx.com/phoneview/trial
> >>
> >> Thanks
> >>
> >> Stephen
> >>
> >> On 5 Nov 2012, at 13:51, "Pawlowski, Adam" <ajp26 at buffalo.edu>
> >>  wrote:
> >>
> >>
> >>
> >> Morning list,
> >>
> >>      From what I can read, TVS has been a thrill ride for those of us
lucky
> enough to run afoul of SBD. I'm hoping we haven't run into such a
situation
> here but I have a question. We just pushed 9.3.1 out to devices from
> 9.1.1SR1, with peer firmware sharing enabled. This went miserably and left
a
> lot of devices stranded at 9.1.1 or at an "Upgrading" screen. Working with
> TAC on that but so far nothing. We began receiving reports from end users
> that their directories were missing, they can't change ringers, etc. Last
time
> this happened we had phones that had picked up a bum ITL from a partial
> rollout, and we had to erase them by hand.
> >>
> >>      This time that shouldn't have been the case - it was just a
firmware
> upgrade. However, it looks like some of the phones that have gone to 9.3.1
> have decided that they want to use IPv6 when talking to the TFTP to verify
> their initial configurations, when they have no TVS server list built on
the
> device. In looking at the phone console logs, you can see that the device
is in
> "IP mode 1" and tries to connect to TFTP, say "192.168.0.1 :: " which is
> obviously not a V6 address. We're not running V6 so it has no address
bound.
> No traffic leaves the phone, but, the phone says it timed out (EAGAIN)
> connecting to the TFTP and won't verify the ITL/CFG/etc.
> >>
> >>       What I'm looking at it is setting V6 to off at the cluster, but,
I don't see
> any way to repair this on affected devices (could be a large amount) other
> than erasing the ITL, or deploying IPv6. Given the leg work that could go
into
> identifying and taking care of these devices, it's arguable as to which
one
> would be harder at this point.
> >>
> >>       Any comment on this with this firmware? Anyone else run into this
> miserable trouble?
> >>
> >> Adam Pawlowski
> >> SUNYAB Network and Classroom Services
> >> _______________________________________________
> >> cisco-voip mailing list
> >> cisco-voip at puck.nether.net
> >> https://puck.nether.net/mailman/listinfo/cisco-voip
> >
> >
> > _______________________________________________
> > cisco-voip mailing list
> > cisco-voip at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-voip
> >
> >
>