[cisco-voip] VCS Expressway upgrade, 8.7 to 12.5

Tue Nov 19 07:57:52 EST 2019

If anything like this farts up on us then I usually remove it from the Expressway and put it back. It doesn’t really cause a ton of harm to do so and seems to clear a lot of issues.

Troubleshooting certificate issues ends up being kind of obnoxious as I’ve yet to find any debugging that really expands on the fault errors that you can see in the event log.

If this were me and I ran into “Decryption Error” and not a validation error, I’d be looking at say the default TLS level and ciphers as they may have changed between 8.7 -> 12.5. I know it’s default TLS 1.2 at this point. I’d actually just changed the s_channel string or what have you to try and eliminate the ciphers flagged for no forward secrecy and it broke MRA for me as well. I got “Internal Server Error” back in Jabber.

I don’t know if it enforces key lengths on the certificates at this point either but that’s where I’d go with this one personally.

Adam

From: cisco-voip <cisco-voip-bounces at puck.nether.net> On Behalf Of Jonathan Charles
Sent: Monday, November 18, 2019 12:24 AM
To: Ryan Huff <ryanhuff at outlook.com>
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] VCS Expressway upgrade, 8.7 to 12.5

So, just an FYI, with MRA down, we rolled back to 8.7.3... going to try this again next Friday...

Should we try going to an intermediate version first, say, 8.11 or something?

Jonathan

On Sun, Nov 17, 2019 at 6:15 PM Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:
Loaded the local certs... no joy...

Jonathan

On Sun, Nov 17, 2019 at 6:07 PM Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>> wrote:
Have you tried adding the IMP identity cert into the Expressway trust? It shouldn’t have to work that way, but if it does, might point to an issue with how the CA chain is being recognized in the trust.

Also, make sure to do a full reboot of the Expressway node after adding certs into the truststore (again, you shouldn’t have to do that but I’ve seen this work before).
Sent from my iPhone

On Nov 17, 2019, at 18:58, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:

When I try to refresh the IMP nodes, I get Failed: Unable to communicate with [[IMPNODE] CryptoError: Decryption failure.

On Sun, Nov 17, 2019 at 5:54 PM Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:
I re-uploaded the root and intermediate CA certificate... still get the same error...

I also tried adding a new AXL user... same error...

Jonathan

On Sun, Nov 17, 2019 at 5:48 PM Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>> wrote:
Likely certificate / trust issues..
Sent from my iPhone

On Nov 17, 2019, at 18:36, Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:

Yep, we are running into clustering issues...

Getting Inactive: (Remote host is reachable but connection is not established. Either refresh this page, or check the credentials.)

For IMP connection, so MRA is down...

Still looking for a fix...

Jonathan

On Fri, Nov 15, 2019 at 7:17 PM Erick Bergquist <erickbee at gmail.com<mailto:erickbee at gmail.com>> wrote:
I’ve done 2 8.11.x to 12.5.5 fine (clustered setup, 4). There is a bug with clustering to watch out for but I did not encounter it. The 12.5 Cisco download page has a note and link about this.

Currently working on jabberd process high memory consumption issue on one node that has been present since 8.11.x which 12.5 had memory leak fix for but still an issue. Slow memory increase over time just on one of the edge nodes.

Going to look over 12.5.6 release notes now....

Erick

On Fri, Nov 15, 2019 at 3:28 PM Matt Jacobson <m4ttjacobson at gmail.com<mailto:m4ttjacobson at gmail.com>> wrote:
If that is the case, then I would double check that it is supported. In the release notes there is a chart for supported platforms based on serial numbers. If it is a legacy Tandberg box, then I suspect 12.x may not work out for you.

On Fri, Nov 15, 2019 at 14:30 Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>> wrote:
This is a legacy Tandberg VCS for video only... no MRA, no remote phones... just inbound and outbound sip video...

Jonathan

On Fri, Nov 15, 2019 at 12:44 PM Pawlowski, Adam <ajp26 at buffalo.edu<mailto:ajp26 at buffalo.edu>> wrote:
We’re at 12.5.3 and probably moving to 12.5.5/12.5.6 somewhere in the Holiday timeframe when everything quiets down a bit.

There hasn’t been really any significant issue upgrading from 8 -> 12, but there have been a couple of bugs that largely are all resolved by deleting and rebuilding whatever the thing is that is misbehaving.

The requirement for the _cup_login and _cisco-uds SRVs went away though it still endlessly logs a warning about not finding them, but it will work.

You do also gain the ability to play with the openssl cipher strings but in my limited experience trying to change those to bump them up a notch, it ends up breaking XMPP or something.

Adam

From: cisco-voip <cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net>> On Behalf Of Jonathan Charles
Sent: Friday, November 15, 2019 11:59 AM
To: Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>>
Cc: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] VCS Expressway upgrade, 8.7 to 12.5

Thanks, the latest is 12.5.6, released last week, I am avoiding it like the plague...and the bug fix doesn't apply to us.

I am going with 12.5.5 (released in August).

I already have release keys (Cisco AM sent them over)...

Hybrid services are on a separate VCS-C that is already 12.5.

My plan is to get new certs if we have any issues

Thanks!

Jonathan

On Fri, Nov 15, 2019 at 10:46 AM Ryan Huff <ryanhuff at outlook.com<mailto:ryanhuff at outlook.com>> wrote:
A couple of thoughts for you...

  *   Get the software release key for 12.x now (you'll be asked to enter it during the upgrade in the GUI). You'll need to work with TAC > GLO for this if (and I assume this would be your case) the existing 8.7 serial is active in Cisco's licensing system. The caveat to trying to do this with Cisco's self-service license re-host tool is that while the 8.7 serial is active, it won't allow you to assign the new 12.x software release PAK to the serial because the serial is already assigned to another software release key.

     *   Take a backup first, your only roll back option is to re-install 8.7 and restore the backup.

  *   Your VMware Hypervisor needs to be 6.0/5/7.

  *   If you have Hybrid Services configured, make sure the management connector is up to date first.

  *   SSL Certificate validation changed a bit in 8.8+

     *   Verify proper forward / reverse DNS for all the relevant touch points
     *   Make sure the Expressway certificate trust is up-to-date with all the current CUCM,CUC,IMP identity certificates (self-signed) or CA certificates (public CA signed certificates).
     *   no duplicate certificates in the Expressway trusts
Beyond that, just pay attention to the caveats list in the upgrade doc for your version of 12.5.x (12.5.4 is the latest I think).

Thanks,

Ryan

________________________________
From: cisco-voip <cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net>> on behalf of Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>>
Sent: Friday, November 15, 2019 10:57 AM
To: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net> <cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>>
Subject: [cisco-voip] VCS Expressway upgrade, 8.7 to 12.5

Can we just upgrade directly or do we need to go to an intermediary version first?

Also, any gotchas besides new certificates?

Jonathan
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=02%7C01%7C%7Ca7812f6d91674afb39ec08d76bba1a5c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637096319353456443&sdata=eAAM2Z%2BXkTR0qvgQFFqqARRdwoAQeWktKEgZjKoUIN8%3D&reserved=0>
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=02%7C01%7C%7Ca7812f6d91674afb39ec08d76bba1a5c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637096319353466452&sdata=5Ssb9MtFJnSWFfkbiH76J5p3s6XRCn977yPgDvRcaBM%3D&reserved=0>
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=02%7C01%7C%7C7cabc92fc21049a2d5fb08d76bb6f0ec%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637096305774892286&sdata=cdlrEIKDc1VPe7FQtAdLT%2FpSn%2FJRQ%2BdqG%2Bv0pvpw7V4%3D&reserved=0<https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=02%7C01%7C%7Ca7812f6d91674afb39ec08d76bba1a5c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637096319353476457&sdata=X5L3fCie69V3nJtmqFL3ZAn02CMAgl0R7ba1Ze%2BXPD0%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20191119/1808ef6a/attachment.htm>