[cisco-voip] CUCM 8.5.1 SU1 to SU3 update failure on CAR db

Mon Jun 4 11:13:16 EDT 2012

Update on this. This is corrected now.  There is a new bug for this,
CSCua24313.

Basically, the Cisco CAR DB service was also not started all the time
(sometimes it was) and developers had to initialize the car db with
the informix tool, then rerun the car installation script to recreate
the database. Then had to create the pid file manually so the cisco
car DB service would start.

Lesson learned, before you do a upgrade/SU patch make sure the CAR DB
service is started and you can get to reports in the CDR
Analysis/Reporting tool web pages as the upgrade uses car even though
you may not ever look at CAR Reports if you're using a 3rd party CDR
reporting tool.

Thanks everyone for the help,
Erick

On Thu, May 17, 2012 at 4:07 PM, Adam Frankel (afrankel)
<afrankel at cisco.com> wrote:
> For 8.x, the query would be a bit different.  The syntax is "run sql car
> select * from tbl_system_preferences".
>
> -Adam
>
>
> ________________________________
> From: Erick B. <erickbee at gmail.com>
> Sent: Thu, May 17, 2012 3:51:29 PM
> To: Wes Sisk <wsisk at cisco.com>
> CC: voip puck <cisco-voip at puck.nether.net>
> Subject: Re: [cisco-voip] CUCM 8.5.1 SU1 to SU3 update failure on CAR db
>
> Wes,
>
> Also a manual back up of CAR works fine, but I can not access any page
> in CDR Tools web page. I get database error page for any menu options
> selected.  All the services are started. When I try to run sql query
> from CLI I get "Database not found or no system permission"  with
> query like run sql select * from car:tbl_system_preferences.
>
>
>
> The following Error(s) occured while processing your request. Contact
> the System Administrator.
>
>    * Click on the 'Close' button, to go to the Main Screen.
>
> Error Code 	Error Description
> 10012 	Database Error. Contact System Administrator.
>
>
> Is there a way to rebuild the CAR Database or have upgrade skip over
> it?  Waiting for callback/reply from TAC.
> It looks like the CAR db is messed up. I hope there is another
> solution besides rebuilding the cluster.
>
> Thanks,
> Erick
>
> On Thu, May 17, 2012 at 10:57 AM, Erick B. <erickbee at gmail.com> wrote:
>> Wes,
>>
>> Backups are good and include CAR Data, all successful and I did a
>> manual backup fine with all options.
>>
>> Haven't ran the recovery disc yet ever on this node, I don't recall
>> instance where the fs went read-only on this in the past. There was
>> SAN issue and CCM/CTI services Core dumped and restarted at same time.
>> The upgrade to SU3 is to get correction for Disk I/O timeout increase
>> from 30sec to 180 sec actually (CSCtj97676).
>>
>> I was going to purge the CAR database and see if that helped.  Is
>> there a way to recreate the CAR db on the linux platform without
>> lengthy TAC involvement? I knew I could do it on Windows.
>>
>> Thanks Wes.
>>
>> On Thu, May 17, 2012 at 10:25 AM, Wes Sisk <wsisk at cisco.com> wrote:
>>> Looks like a couple of things going on.  Migrating CAR data took
>>> significant time and likely timed out.  That will slow the upgrade. Toss the
>>> CAR data unless you really need it.
>>>
>>> Then you hit:
>>> "Primary and Mirror chunks are bad". We have several of these on record
>>> where the consensus seems to be: "Database restore failed because of bad
>>> chunks in active partition."
>>>
>>> Has this sever ever suffered an unexpected reboot?  Look for it using
>>> tips from:
>>> CSCtr88859    add alarm and alert for unexpected reboot
>>>
>>> Has recovery disk been used with filesystem repair? Note that fs repair
>>> only truncates bad files. Thus:
>>> CSCth60800    Recovery Disc warning to rebuild system after file system
>>> repair
>>>
>>>
>>> I am not clear if the existing in-place database is bad or if it is an
>>> export or newly created database that is bad. If the former that will be a
>>> big problem. If the later then it suggests a problem with the filesystem and
>>> potentially hardware.  Checking and repairing those may be sufficient.
>>>
>>> To get a better idea check recent backups. Are backups successful?
>>> Backups have to dump the entire database. Does the backup include the CAR
>>> database? Can you take a full backup?
>>>
>>> Regards,
>>> Wes
>>>
>>>
>>>
>>> On May 17, 2012, at 10:08 AM, Erick B. wrote:
>>>
>>> Does anyone know of any update issues from 8.5.1 SU1 to SU3?  I've
>>> done some looking, hot RSS feeds, etc and not finding anything yet and
>>> am working with TAC on this but just thought I'd ask here.
>>>
>>> Upgrade patch ran fine for about an hour, then gave error below. I did
>>> the update via SFTP and the file size and MD5 matched.
>>>
>>> Error encountered: An unknown error occurred while accessing the upgrade
>>> file
>>>
>>> This is snippet from install logs, it is fine before this and appears
>>> to have an issue with the CAR database.
>>>
>>>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) run command:
>>> /usr/local/cm/bin/pryftool --system cm --hive dbl --set
>>> current_informixserver="cm1_car8_5_1_13900_5"|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Successfully added
>>> current_informixserver in dbl prefs.SET
>>> prior_informixserver|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Inside Linux|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,721 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:529) - Exception in cm_car_post_time
>>> script|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,740 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:530) - Following things might have
>>> happened|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,746 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:531) - 1.User might have set the time to 60
>>> mins|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,747 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:532) - 2.DMABackupInfo.inf file or CARMigraionTime
>>> entry may not be present |<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,747 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:533) - Hence setting the default migration time
>>> |<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,748 INFO  [main] migration.MigrationTool
>>> (MigrationTool.java:108) - main()-->|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:35
>>> CDT 2012:main:connect() SQLException|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:35
>>> CDT 2012:main:|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,996 WARN  [main] dbl.Connector (Connector.java:1184) -
>>> Connector(NOT CONNECTED:
>>> Driver=com.informix.jdbc.IfxDriver;)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) java.sql.SQLException:
>>> Cannot open system catalog (systables).|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.a(IfxSqli.java:3457)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3774)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2580)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2496)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.informix.jdbc.IfxSqli.executeOpenDatabase(IfxSqli.java:2021)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1622)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxDriver.connect(IfxDriver.java:254)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.sql.DriverManager.getConnection(DriverManager.java:582)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.sql.DriverManager.getConnection(DriverManager.java:207)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.getConn(Connector.java:769)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.connect(Connector.java:677)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.connect(Connector.java:619)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:978)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:964)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.car.beans.EBSystemPreferences.loadCurrentData(EBSystemPreferences.java:171)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.car.beans.EBSystemPreferences.<init>(EBSystemPreferences.java:98)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.car.database.SchemaCreator.<clinit>(SchemaCreator.java:111)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.car.migration.MigrationTool.main(MigrationTool.java:110)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Caused by:
>>> java.sql.SQLException: ISAM error: Primary and Mirror chunks are
>>> bad|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:408)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3779)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         ... 20
>>> more|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:36
>>> CDT 2012:main:handleConnectionException: ErrorCode is
>>> -311|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:36,007 WARN  [main] dbl.Connector (Connector.java:1187) -
>>> handleConnectionException: ErrorCode is -311|<LVL::Debug>
>>>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:36,007 ERROR [main] beans.EBSystemPreferences
>>> (EBSystemPreferences.java:199) - loadCurrentData(): Error while
>>> loading Sys Pref data|<LVL::Debug>
>>>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) java.sql.SQLException:
>>> Cannot open system catalog (systables).|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.a(IfxSqli.java:3457)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3774)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2580)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2496)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.informix.jdbc.IfxSqli.executeOpenDatabase(IfxSqli.java:2021)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1622)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxDriver.connect(IfxDriver.java:254)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.sql.DriverManager.getConnection(DriverManager.java:582)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.sql.DriverManager.getConnection(DriverManager.java:207)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.getConn(Connector.java:769)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.connect(Connector.java:677)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.connect(Connector.java:619)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:978)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:964)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.car.beans.EBSystemPreferences.loadCurrentData(EBSystemPreferences.java:171)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.car.beans.EBSystemPreferences.<init>(EBSystemPreferences.java:98)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.car.database.SchemaCreator.<clinit>(SchemaCreator.java:111)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.cisco.ccm.car.migration.MigrationTool.main(MigrationTool.java:110)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Caused by:
>>> java.sql.SQLException: ISAM error: Primary and Mirror chunks are
>>> bad|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>>
>>> com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:408)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3779)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         ... 20
>>> more|<LVL::Debug>
>>>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:36,008 ERROR [main] database.SchemaCreator
>>> (SchemaCreator.java:128) - error while setting the manual purge status
>>> to 0 |<LVL::Debug>
>>> 05/17/2012
>>> _______________________________________________
>>> cisco-voip mailing list
>>> cisco-voip at puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>
>>>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
>
>