[cisco-voip] Issues with certificate security following upgrade to 8.6

[Nathan Reeves]

Upgraded our cluster from 8.0.3 to 8.6.1 last weekend.  Upgrade
completed successfully (took a while but given it was a major jump I
expected that).  Everything appeared to be working fine but come
Monday I started getting calls from users who couldn't use the
corporate directory on their phones.  When they attempted to access
it, a 'host not found' error occurred.  Ended up tracing this back to
the phones being unable to access https based links due to a tvs /
certain issue.

At this point, the manual deletion of the phones security tlv file
fixes the issue.  I think i've got about 100 phones affected, so the
process isn't too bad to get done, just going to be time consuming.

What I want to try and work out is what I've done wrong in the upgrade
process which has caused this issue.  Basic overview of the steps
taken is as follows:
 - upgraded pub to 8.6.1.  Booted back into 8.0.3 following upgrade.
 - upgraded sub to 8.6.1.  For some reason, even though I marked not
to boot to the upgraded partition, it booted into 8.6.1.
 - booted pub into 8.6.1.
 - waited for phones to settle down. Eg from switchback to sub and
firmware upgrades etc.
 - updated servers to use dns (previously dns was not set).
 - checked that db replication was successful (saw some issues which a
reboot and updating reverse dns entries appeared to fix.)
 - 48 hours later installed new tomcat certs which had been signed by
our windows domain CA.

In troubleshooting the cert issue on the phones, I can see it heads to
the tvs to do a cert verify when it tries to access ssl links and
secured tftp.  On the phones I can see it send a request to the tvs
and it lists a certificate serial which I'm assuming it is trying to
verify.  The request then fails and the phone fails to load the
cnf.XML etc.  If I check the two servers, both have a certificate
which matches the serial listed on the phone when it is requesting the
tvs to check the cert.

I saw the previus issue with 8.5.1 and the tftp tlv file failing to be
written but it's a different issue to that.  In looking at the tvs
logs, I do see a file failed to be written but with an error -2.  The
file was ctlfile.tlv.

Have I caused this issue with the update of the dns settings on the
servers and a possible regeneration of the certifates at this point.
Was there a process I should have followed.

I'm considering contacting tac in the morning to see if theres
anything obvious I've done wrong and if there's a way to resolve the
issue without touching phones manually, but if it's obvious to someone
here what I've screwed up it would be great to hear it.

Thanks

Nathan