[cisco-voip] CUCM 8.5.1 SU1 to SU3 update failure on CAR db

Wes Sisk wsisk at cisco.com
Thu May 17 16:02:03 EDT 2012 

Time for a TAC case. Either the database is gone or the pointers to it are incorrect ala
CSCsm01916    Cannot generate CAR reports - getting db errors returned in the message

either way TAC and development are required to correct pointers or recreate database
/wes

On May 17, 2012, at 3:51 PM, Erick B. wrote:

Wes,

Also a manual back up of CAR works fine, but I can not access any page
in CDR Tools web page. I get database error page for any menu options
selected.  All the services are started. When I try to run sql query
from CLI I get "Database not found or no system permission"  with
query like run sql select * from car:tbl_system_preferences.



The following Error(s) occured while processing your request. Contact
the System Administrator.

  * Click on the 'Close' button, to go to the Main Screen.

Error Code 	Error Description
10012 	Database Error. Contact System Administrator.


Is there a way to rebuild the CAR Database or have upgrade skip over
it?  Waiting for callback/reply from TAC.
It looks like the CAR db is messed up. I hope there is another
solution besides rebuilding the cluster.

Thanks,
Erick

On Thu, May 17, 2012 at 10:57 AM, Erick B. <erickbee at gmail.com> wrote:
> Wes,
> 
> Backups are good and include CAR Data, all successful and I did a
> manual backup fine with all options.
> 
> Haven't ran the recovery disc yet ever on this node, I don't recall
> instance where the fs went read-only on this in the past. There was
> SAN issue and CCM/CTI services Core dumped and restarted at same time.
> The upgrade to SU3 is to get correction for Disk I/O timeout increase
> from 30sec to 180 sec actually (CSCtj97676).
> 
> I was going to purge the CAR database and see if that helped.  Is
> there a way to recreate the CAR db on the linux platform without
> lengthy TAC involvement? I knew I could do it on Windows.
> 
> Thanks Wes.
> 
> On Thu, May 17, 2012 at 10:25 AM, Wes Sisk <wsisk at cisco.com> wrote:
>> Looks like a couple of things going on.  Migrating CAR data took significant time and likely timed out.  That will slow the upgrade. Toss the CAR data unless you really need it.
>> 
>> Then you hit:
>> "Primary and Mirror chunks are bad". We have several of these on record where the consensus seems to be: "Database restore failed because of bad chunks in active partition."
>> 
>> Has this sever ever suffered an unexpected reboot?  Look for it using tips from:
>> CSCtr88859    add alarm and alert for unexpected reboot
>> 
>> Has recovery disk been used with filesystem repair? Note that fs repair only truncates bad files. Thus:
>> CSCth60800    Recovery Disc warning to rebuild system after file system repair
>> 
>> 
>> I am not clear if the existing in-place database is bad or if it is an export or newly created database that is bad. If the former that will be a big problem. If the later then it suggests a problem with the filesystem and potentially hardware.  Checking and repairing those may be sufficient.
>> 
>> To get a better idea check recent backups. Are backups successful? Backups have to dump the entire database. Does the backup include the CAR database? Can you take a full backup?
>> 
>> Regards,
>> Wes
>> 
>> 
>> 
>> On May 17, 2012, at 10:08 AM, Erick B. wrote:
>> 
>> Does anyone know of any update issues from 8.5.1 SU1 to SU3?  I've
>> done some looking, hot RSS feeds, etc and not finding anything yet and
>> am working with TAC on this but just thought I'd ask here.
>> 
>> Upgrade patch ran fine for about an hour, then gave error below. I did
>> the update via SFTP and the file size and MD5 matched.
>> 
>> Error encountered: An unknown error occurred while accessing the upgrade file
>> 
>> This is snippet from install logs, it is fine before this and appears
>> to have an issue with the CAR database.
>> 
>> 
>> 05/17/2012 00:33:36 component_install|(CAPTURE) run command:
>> /usr/local/cm/bin/pryftool --system cm --hive dbl --set
>> current_informixserver="cm1_car8_5_1_13900_5"|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) Successfully added
>> current_informixserver in dbl prefs.SET
>> prior_informixserver|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) Inside Linux|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:35,721 WARN  [main] migration.MigrationTool
>> (MigrationTool.java:529) - Exception in cm_car_post_time
>> script|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:35,740 WARN  [main] migration.MigrationTool
>> (MigrationTool.java:530) - Following things might have
>> happened|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:35,746 WARN  [main] migration.MigrationTool
>> (MigrationTool.java:531) - 1.User might have set the time to 60
>> mins|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:35,747 WARN  [main] migration.MigrationTool
>> (MigrationTool.java:532) - 2.DMABackupInfo.inf file or CARMigraionTime
>> entry may not be present |<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:35,747 WARN  [main] migration.MigrationTool
>> (MigrationTool.java:533) - Hence setting the default migration time
>> |<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:35,748 INFO  [main] migration.MigrationTool
>> (MigrationTool.java:108) - main()-->|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:35
>> CDT 2012:main:connect() SQLException|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:35
>> CDT 2012:main:|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:35,996 WARN  [main] dbl.Connector (Connector.java:1184) -
>> Connector(NOT CONNECTED:
>> Driver=com.informix.jdbc.IfxDriver;)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) java.sql.SQLException:
>> Cannot open system catalog (systables).|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.a(IfxSqli.java:3457)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3774)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2580)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2496)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.executeOpenDatabase(IfxSqli.java:2021)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1622)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxDriver.connect(IfxDriver.java:254)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> java.sql.DriverManager.getConnection(DriverManager.java:582)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> java.sql.DriverManager.getConnection(DriverManager.java:207)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.getConn(Connector.java:769)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.connect(Connector.java:677)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.connect(Connector.java:619)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:978)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:964)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.car.beans.EBSystemPreferences.loadCurrentData(EBSystemPreferences.java:171)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.car.beans.EBSystemPreferences.<init>(EBSystemPreferences.java:98)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.car.database.SchemaCreator.<clinit>(SchemaCreator.java:111)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.car.migration.MigrationTool.main(MigrationTool.java:110)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) Caused by:
>> java.sql.SQLException: ISAM error: Primary and Mirror chunks are
>> bad|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:408)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3779)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         ... 20 more|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:36
>> CDT 2012:main:handleConnectionException: ErrorCode is
>> -311|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:36,007 WARN  [main] dbl.Connector (Connector.java:1187) -
>> handleConnectionException: ErrorCode is -311|<LVL::Debug>
>> 
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:36,007 ERROR [main] beans.EBSystemPreferences
>> (EBSystemPreferences.java:199) - loadCurrentData(): Error while
>> loading Sys Pref data|<LVL::Debug>
>> 
>> 05/17/2012 00:33:36 component_install|(CAPTURE) java.sql.SQLException:
>> Cannot open system catalog (systables).|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.a(IfxSqli.java:3457)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3774)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2580)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2496)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.executeOpenDatabase(IfxSqli.java:2021)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1622)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> java.lang.reflect.Constructor.newInstance(Constructor.java:513)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxDriver.connect(IfxDriver.java:254)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> java.sql.DriverManager.getConnection(DriverManager.java:582)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> java.sql.DriverManager.getConnection(DriverManager.java:207)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.getConn(Connector.java:769)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.connect(Connector.java:677)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.connect(Connector.java:619)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:978)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:964)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.car.beans.EBSystemPreferences.loadCurrentData(EBSystemPreferences.java:171)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.car.beans.EBSystemPreferences.<init>(EBSystemPreferences.java:98)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.car.database.SchemaCreator.<clinit>(SchemaCreator.java:111)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.cisco.ccm.car.migration.MigrationTool.main(MigrationTool.java:110)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE) Caused by:
>> java.sql.SQLException: ISAM error: Primary and Mirror chunks are
>> bad|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:408)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         at
>> com.informix.jdbc.IfxSqli.E(IfxSqli.java:3779)|<LVL::Debug>
>> 05/17/2012 00:33:36 component_install|(CAPTURE)         ... 20 more|<LVL::Debug>
>> 
>> 05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
>> 00:33:36,008 ERROR [main] database.SchemaCreator
>> (SchemaCreator.java:128) - error while setting the manual purge status
>> to 0 |<LVL::Debug>
>> 05/17/2012
>> _______________________________________________
>> cisco-voip mailing list
>> cisco-voip at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-voip
>> 
>>

More information about the cisco-voip mailing list