[cisco-voip] CUCM 8.5.1 SU1 to SU3 update failure on CAR db

Thu May 17 16:20:21 EDT 2012

Wes, Thanks. I'm taking this offline for now.

On Thu, May 17, 2012 at 3:02 PM, Wes Sisk <wsisk at cisco.com> wrote:
> Time for a TAC case. Either the database is gone or the pointers to it are incorrect ala
> CSCsm01916    Cannot generate CAR reports - getting db errors returned in the message
>
> either way TAC and development are required to correct pointers or recreate database
> /wes
>
> On May 17, 2012, at 3:51 PM, Erick B. wrote:
>
> Wes,
>
> Also a manual back up of CAR works fine, but I can not access any page
> in CDR Tools web page. I get database error page for any menu options
> selected.  All the services are started. When I try to run sql query
> from CLI I get "Database not found or no system permission"  with
> query like run sql select * from car:tbl_system_preferences.
>
>
>
> The following Error(s) occured while processing your request. Contact
> the System Administrator.
>
>  * Click on the 'Close' button, to go to the Main Screen.
>
> Error Code      Error Description
> 10012   Database Error. Contact System Administrator.
>
>
> Is there a way to rebuild the CAR Database or have upgrade skip over
> it?  Waiting for callback/reply from TAC.
> It looks like the CAR db is messed up. I hope there is another
> solution besides rebuilding the cluster.
>
> Thanks,
> Erick
>
> On Thu, May 17, 2012 at 10:57 AM, Erick B. <erickbee at gmail.com> wrote:
>> Wes,
>>
>> Backups are good and include CAR Data, all successful and I did a
>> manual backup fine with all options.
>>
>> Haven't ran the recovery disc yet ever on this node, I don't recall
>> instance where the fs went read-only on this in the past. There was
>> SAN issue and CCM/CTI services Core dumped and restarted at same time.
>> The upgrade to SU3 is to get correction for Disk I/O timeout increase
>> from 30sec to 180 sec actually (CSCtj97676).
>>
>> I was going to purge the CAR database and see if that helped.  Is
>> there a way to recreate the CAR db on the linux platform without
>> lengthy TAC involvement? I knew I could do it on Windows.
>>
>> Thanks Wes.
>>
>> On Thu, May 17, 2012 at 10:25 AM, Wes Sisk <wsisk at cisco.com> wrote:
>>> Looks like a couple of things going on.  Migrating CAR data took significant time and likely timed out.  That will slow the upgrade. Toss the CAR data unless you really need it.
>>>
>>> Then you hit:
>>> "Primary and Mirror chunks are bad". We have several of these on record where the consensus seems to be: "Database restore failed because of bad chunks in active partition."
>>>
>>> Has this sever ever suffered an unexpected reboot?  Look for it using tips from:
>>> CSCtr88859    add alarm and alert for unexpected reboot
>>>
>>> Has recovery disk been used with filesystem repair? Note that fs repair only truncates bad files. Thus:
>>> CSCth60800    Recovery Disc warning to rebuild system after file system repair
>>>
>>>
>>> I am not clear if the existing in-place database is bad or if it is an export or newly created database that is bad. If the former that will be a big problem. If the later then it suggests a problem with the filesystem and potentially hardware.  Checking and repairing those may be sufficient.
>>>
>>> To get a better idea check recent backups. Are backups successful? Backups have to dump the entire database. Does the backup include the CAR database? Can you take a full backup?
>>>
>>> Regards,
>>> Wes
>>>
>>>
>>>
>>> On May 17, 2012, at 10:08 AM, Erick B. wrote:
>>>
>>> Does anyone know of any update issues from 8.5.1 SU1 to SU3?  I've
>>> done some looking, hot RSS feeds, etc and not finding anything yet and
>>> am working with TAC on this but just thought I'd ask here.
>>>
>>> Upgrade patch ran fine for about an hour, then gave error below. I did
>>> the update via SFTP and the file size and MD5 matched.
>>>
>>> Error encountered: An unknown error occurred while accessing the upgrade file
>>>
>>> This is snippet from install logs, it is fine before this and appears
>>> to have an issue with the CAR database.
>>>
>>>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) run command:
>>> /usr/local/cm/bin/pryftool --system cm --hive dbl --set
>>> current_informixserver="cm1_car8_5_1_13900_5"|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Successfully added
>>> current_informixserver in dbl prefs.SET
>>> prior_informixserver|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Inside Linux|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,721 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:529) - Exception in cm_car_post_time
>>> script|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,740 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:530) - Following things might have
>>> happened|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,746 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:531) - 1.User might have set the time to 60
>>> mins|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,747 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:532) - 2.DMABackupInfo.inf file or CARMigraionTime
>>> entry may not be present |<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,747 WARN  [main] migration.MigrationTool
>>> (MigrationTool.java:533) - Hence setting the default migration time
>>> |<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,748 INFO  [main] migration.MigrationTool
>>> (MigrationTool.java:108) - main()-->|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:35
>>> CDT 2012:main:connect() SQLException|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:35
>>> CDT 2012:main:|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:35,996 WARN  [main] dbl.Connector (Connector.java:1184) -
>>> Connector(NOT CONNECTED:
>>> Driver=com.informix.jdbc.IfxDriver;)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) java.sql.SQLException:
>>> Cannot open system catalog (systables).|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.a(IfxSqli.java:3457)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3774)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2580)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2496)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.executeOpenDatabase(IfxSqli.java:2021)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1622)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxDriver.connect(IfxDriver.java:254)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.sql.DriverManager.getConnection(DriverManager.java:582)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.sql.DriverManager.getConnection(DriverManager.java:207)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.getConn(Connector.java:769)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.connect(Connector.java:677)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.connect(Connector.java:619)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:978)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:964)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.car.beans.EBSystemPreferences.loadCurrentData(EBSystemPreferences.java:171)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.car.beans.EBSystemPreferences.<init>(EBSystemPreferences.java:98)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.car.database.SchemaCreator.<clinit>(SchemaCreator.java:111)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.car.migration.MigrationTool.main(MigrationTool.java:110)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Caused by:
>>> java.sql.SQLException: ISAM error: Primary and Mirror chunks are
>>> bad|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:408)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3779)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         ... 20 more|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:36
>>> CDT 2012:main:handleConnectionException: ErrorCode is
>>> -311|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:36,007 WARN  [main] dbl.Connector (Connector.java:1187) -
>>> handleConnectionException: ErrorCode is -311|<LVL::Debug>
>>>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:36,007 ERROR [main] beans.EBSystemPreferences
>>> (EBSystemPreferences.java:199) - loadCurrentData(): Error while
>>> loading Sys Pref data|<LVL::Debug>
>>>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) java.sql.SQLException:
>>> Cannot open system catalog (systables).|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.a(IfxSqli.java:3457)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3774)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2580)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2496)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.executeOpenDatabase(IfxSqli.java:2021)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1622)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxDriver.connect(IfxDriver.java:254)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.sql.DriverManager.getConnection(DriverManager.java:582)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> java.sql.DriverManager.getConnection(DriverManager.java:207)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.getConn(Connector.java:769)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.connect(Connector.java:677)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.connect(Connector.java:619)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:978)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:964)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.car.beans.EBSystemPreferences.loadCurrentData(EBSystemPreferences.java:171)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.car.beans.EBSystemPreferences.<init>(EBSystemPreferences.java:98)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.car.database.SchemaCreator.<clinit>(SchemaCreator.java:111)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.cisco.ccm.car.migration.MigrationTool.main(MigrationTool.java:110)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) Caused by:
>>> java.sql.SQLException: ISAM error: Primary and Mirror chunks are
>>> bad|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:408)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3779)|<LVL::Debug>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE)         ... 20 more|<LVL::Debug>
>>>
>>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>>> 00:33:36,008 ERROR [main] database.SchemaCreator
>>> (SchemaCreator.java:128) - error while setting the manual purge status
>>> to 0 |<LVL::Debug>
>>> 05/17/2012
>>> _______________________________________________
>>> cisco-voip mailing list
>>> cisco-voip at puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>
>>>
>
>