[cisco-voip] UCCX 11.5 Upgrade Disaster -- SOLVED

[Ayoub,Gregory]

Just in case anyone is curious, the cause of our 11.5 IPCC failure was finally solved by Cisco TAC.

The license file issued to us after the upgrade did not contain proper licensing for outbound dialing.   Because there was no outbound licensing, and because the campaign was carried forward by the upgrade from 10.6, it was still able to run even though it appeared as if the outbound campaign was not installed (you have to have outbound license to even see the outbound campaign menu).   When the outbound campaign kicked off, this caused a lock on all of the inbound CTI ports which resulted in the busy signals.   Because of the severity of this failure scenario, a bug ID was created to track resolution so the outbound campaign will check for proper licensing before starting.

The bright side was that my guess as to the source of the failure was correct :-)

From: Ayoub,Gregory
Sent: Monday, November 14, 2016 2:53 PM
To: cisco-voip at puck.nether.net
Subject: UCCX 11.5 Upgrade Disaster

We recently attempted to upgrade UCCX 10.5 -- > 11.5.  The deployment was HA, and we didn't see much risk.   The upgrade went fairly smooth, and we hit a finesse bug which required the ECDSA COP file.  While minor, it's still not mentioned in the release notes.

The real shocker was 4 hours later when the system stopped accepting calls and just handed out fast busys.  Failing over to the secondary would fix the issue for a few minutes, but then fast busy.

Our entire contact center, which is HA, was entirely down.  Primary down, Secondary Down, and TAC was unable to resolve after hours and hours.  It was a total unmitigated Cisco disaster.

Rolling back fixed the problem.  And then rolling forward again to 11.5 the system worked great again - but only for 4 hours.  Then endless fast busys.  We rolled back and are on 10.5 working fine.  But we are at a loss what could be causing this problem.  Cisco TAC is in the same boat.

If I had to guess, that seems like more of a licensing failure, because TAC even tried replacing our license.   Anyone have a similar experience?

Thanks Greg.




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20170118/ea3bfa01/attachment.html>