[cisco-voip] Fun message on CER after power outage...

Wed Feb 23 08:21:14 EST 2022

We rebuilt a bunch of them after a data center failure over summer. It did take up a bunch of time to schedule that maintenance and have someone run through it, but that was the worst of it.

In our lab things have been powered off / on forcefully before, and never any ill effect that I've run into that I can say is direct result, other than the experience I had with CER.

This document has been around for a number of years:

https://www.cisco.com/c/en/us/support/docs/unified-communications/unified-communications-manager-callmanager/116717-trouble-cucm-shutdown-00.html

so I would imagine that TAC, especially now with a shortage of engineers/experience, might want to check that instead of chasing weird issues.
We'd run into things where phone tracking would just stop operating, even with the process running. Or it would say it ran and nothing happened or updated until we restarted it.
Since CER is such jank in the background, with little exposed to really play with or look at, it was way faster to export, backup, re-install, restore, than to grind through that with TAC - and it seemed to solve our problem.

That being said restoring from backup is acceptable in these cases, and I have to wonder if it's possible for data in the database to be bad/corrupted as well in the backup if it's taken after that message is displayed. They didn't care. I think the call was some sort of consistency checking instead of just asking for it to be rebuilt, but I don't know if that would be delivered.

CER documentation specifically seemed to have a lot more wording in it about making sure it is running, it's working well, backed up, etc due to the nature of the service it is providing - likely to try and hedge against some sort of blame being placed on Cisco if an emergency call didn't complete.

Best,

Adam

From: Jonathan Charles <jonvoip at gmail.com>
Sent: Tuesday, February 22, 2022 5:35 PM
To: Lelio Fulgenzi <lelio at uoguelph.ca>
Cc: Nick Russo <russon81 at yahoo.com>; cisco-voip at puck.nether.net; Adam Pawlowski <ajp26 at buffalo.edu>
Subject: Re: [cisco-voip] Fun message on CER after power outage...

Yeah, that seems a bit extreme...

Everything seems fine with the box... every check I have made says it is working fine... (DB Replication, status, test calls, phone tracking...)

Jonathan

On Tue, Feb 22, 2022 at 3:11 PM Lelio Fulgenzi <lelio at uoguelph.ca<mailto:lelio at uoguelph.ca>> wrote:
A UPS with comms available. In our datacenter we have a UPS but it's not comms enabled.

This is yet another "special requirement" for Cisco collab software that will drive people crazy. Including management.

"Sorry, you took power away, I know have to spend the weekend rebuilding 8 servers (or more)"

"But no one else is saying they have to rebuild servers"

"Yeah, it's another one of those special requirements"

From: Nick Russo <russon81 at yahoo.com<mailto:russon81 at yahoo.com>>
Sent: Tuesday, February 22, 2022 4:07 PM
To: Lelio Fulgenzi <lelio at uoguelph.ca<mailto:lelio at uoguelph.ca>>; Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>>; cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>; Adam Pawlowski <ajp26 at buffalo.edu<mailto:ajp26 at buffalo.edu>>
Subject: Re: [cisco-voip] Fun message on CER after power outage...

CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp at uoguelph.ca<mailto:IThelp at uoguelph.ca>

This is my favorite new Cisco "feature".  All the 14.x platforms have this.  I think they implemented in 12.5(su4 or 5), but if the system goes down dirty, the official Cisco policy is to do a restore from backup.  So far, I haven't had any actual issues after this happens, but if your system is in this state, TAC won't help you until you've rebuilt.  There are a few options for graceful shutdown in VMWare, but your server needs to be on a UPS for them to work.

On Tuesday, February 22, 2022, 01:01:39 PM PST, Adam Pawlowski <ajp26 at buffalo.edu<mailto:ajp26 at buffalo.edu>> wrote:

I need to find it, but of course can't right now, but I swear this is in the ER documentation somewhere, some version, that the system must be re-installed if it shuts down unexpectedly. Outside of this message.

I got called out on it by TAC, and they have documentation way preceding that message appearing visibly to point out where unexpected shutdowns have occurred.

There was a back and forth in the community on this, with a lot of people more interesting in making the message go away than trying to figure out if the system was okay or rebuild it.

I guess in some places it is normal to operate the infrastructure on car batteries or what have you. I really must be missing something as this feels like the old "put tape over the check engine light" .

Same thing I said there - personally, rebuild the thing when you can on your time table, instead of having it fail later, or during an upgrade.

CER seems to be the easiest thing to re-install - if you have exports you can replace the system functionality rather quickly even when DRF backups are no good.

Adam

From: cisco-voip <cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net>> On Behalf Of Lelio Fulgenzi
Sent: Tuesday, February 22, 2022 3:07 PM
To: Jonathan Charles <jonvoip at gmail.com<mailto:jonvoip at gmail.com>>; cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: Re: [cisco-voip] Fun message on CER after power outage...

Yes - those warnings were making their way around another forum. It all has to do with reviewing the log files for a corresponding manual shutdown for any bootup sequences.

There is a fix (?) for the core apps found here. Normally these should work on CER but it's not listed. Not sure why.

https://www.cisco.com/web/software/286319173/139477/ciscocm.add_utils_ungraceful_warn_disable_v1.0.cop-README.pdf<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cisco.com%2Fweb%2Fsoftware%2F286319173%2F139477%2Fciscocm.add_utils_ungraceful_warn_disable_v1.0.cop-README.pdf&data=04%7C01%7Cajp26%40buffalo.edu%7C9c8a6142dfee47add90a08d9f6539cca%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C637811663165959984%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ib%2Bo%2BoEy%2Ff6WGI6SiZmjpnVBevfzDl63KmD7vL1nioM%3D&reserved=0>

>From the readme (above):

The fix is included natively in 12.5.1SU6 (12.5.1.16900-x) and higher and 14SU2 (14.0.1.12900-x) and higher.

Note: I don't think this fixes the issue, just removes the warning. If TAC reviews the logs and finds an ungraceful shutdown, they can start asking you to rebuild.

PS I don't see why a system can't recover without a rebuild either.

From: cisco-voip <cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net>> On Behalf Of Jonathan Charles
Sent: Tuesday, February 22, 2022 2:58 PM
To: cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
Subject: [cisco-voip] Fun message on CER after power outage...

CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp at uoguelph.ca<mailto:IThelp at uoguelph.ca>

Only on the CLI...

WARNING: Ungraceful shutdown detected - A rebuild of this node is highly recommended to ensure no negative impact(such as configuration or file system corruption). For rebuild instructions, see the installation guide.

The bugs suggest it should only be on 12.5 and early... this is on CER 14...

Any recommendations? This seems a bit extreme...

The system appears to be working normally.

Jonathan
_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=04%7C01%7Cajp26%40buffalo.edu%7C9c8a6142dfee47add90a08d9f6539cca%7C96464a8af8ed40b199e25f6b50a20250%7C0%7C0%7C637811663165959984%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Uxjjyb2ebwKjxw7Bp3qCE6oL%2F37sPMWoSoX55hC5M5A%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20220223/7f2b580a/attachment.htm>