[cisco-voip] CUCM 8.5.1 SU1 to SU3 update failure on CAR db

Thu May 17 17:07:26 EDT 2012

For 8.x, the query would be a bit different.  The syntax is "run sql car 
select * from tbl_system_preferences".

-Adam

------------------------------------------------------------------------
*From:* Erick B. <erickbee at gmail.com>
*Sent:* Thu, May 17, 2012 3:51:29 PM
*To:* Wes Sisk <wsisk at cisco.com>
*CC:* voip puck <cisco-voip at puck.nether.net>
*Subject:* Re: [cisco-voip] CUCM 8.5.1 SU1 to SU3 update failure on CAR db

> Wes,
>
> Also a manual back up of CAR works fine, but I can not access any page
> in CDR Tools web page. I get database error page for any menu options
> selected.  All the services are started. When I try to run sql query
> from CLI I get "Database not found or no system permission"  with
> query like run sql select * from car:tbl_system_preferences.
>
>
>
> The following Error(s) occured while processing your request. Contact
> the System Administrator.
>
>     * Click on the 'Close' button, to go to the Main Screen.
>
> Error Code 	Error Description
> 10012 	Database Error. Contact System Administrator.
>
>
> Is there a way to rebuild the CAR Database or have upgrade skip over
> it?  Waiting for callback/reply from TAC.
> It looks like the CAR db is messed up. I hope there is another
> solution besides rebuilding the cluster.
>
> Thanks,
> Erick
>
> On Thu, May 17, 2012 at 10:57 AM, Erick B.<erickbee at gmail.com>  wrote:
> >  Wes,
> >
> >  Backups are good and include CAR Data, all successful and I did a
> >  manual backup fine with all options.
> >
> >  Haven't ran the recovery disc yet ever on this node, I don't recall
> >  instance where the fs went read-only on this in the past. There was
> >  SAN issue and CCM/CTI services Core dumped and restarted at same time.
> >  The upgrade to SU3 is to get correction for Disk I/O timeout increase
> >  from 30sec to 180 sec actually (CSCtj97676).
> >
> >  I was going to purge the CAR database and see if that helped.  Is
> >  there a way to recreate the CAR db on the linux platform without
> >  lengthy TAC involvement? I knew I could do it on Windows.
> >
> >  Thanks Wes.
> >
> >  On Thu, May 17, 2012 at 10:25 AM, Wes Sisk<wsisk at cisco.com>  wrote:
> >>  Looks like a couple of things going on.  Migrating CAR data took significant time and likely timed out.  That will slow the upgrade. Toss the CAR data unless you really need it.
> >>
> >>  Then you hit:
> >>  "Primary and Mirror chunks are bad". We have several of these on record where the consensus seems to be: "Database restore failed because of bad chunks in active partition."
> >>
> >>  Has this sever ever suffered an unexpected reboot?  Look for it using tips from:
> >>  CSCtr88859    add alarm and alert for unexpected reboot
> >>
> >>  Has recovery disk been used with filesystem repair? Note that fs repair only truncates bad files. Thus:
> >>  CSCth60800    Recovery Disc warning to rebuild system after file system repair
> >>
> >>
> >>  I am not clear if the existing in-place database is bad or if it is an export or newly created database that is bad. If the former that will be a big problem. If the later then it suggests a problem with the filesystem and potentially hardware.  Checking and repairing those may be sufficient.
> >>
> >>  To get a better idea check recent backups. Are backups successful? Backups have to dump the entire database. Does the backup include the CAR database? Can you take a full backup?
> >>
> >>  Regards,
> >>  Wes
> >>
> >>
> >>
> >>  On May 17, 2012, at 10:08 AM, Erick B. wrote:
> >>
> >>  Does anyone know of any update issues from 8.5.1 SU1 to SU3?  I've
> >>  done some looking, hot RSS feeds, etc and not finding anything yet and
> >>  am working with TAC on this but just thought I'd ask here.
> >>
> >>  Upgrade patch ran fine for about an hour, then gave error below. I did
> >>  the update via SFTP and the file size and MD5 matched.
> >>
> >>  Error encountered: An unknown error occurred while accessing the upgrade file
> >>
> >>  This is snippet from install logs, it is fine before this and appears
> >>  to have an issue with the CAR database.
> >>
> >>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) run command:
> >>  /usr/local/cm/bin/pryftool --system cm --hive dbl --set
> >>  current_informixserver="cm1_car8_5_1_13900_5"|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) Successfully added
> >>  current_informixserver in dbl prefs.SET
> >>  prior_informixserver|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) Inside Linux|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:35,721 WARN  [main] migration.MigrationTool
> >>  (MigrationTool.java:529) - Exception in cm_car_post_time
> >>  script|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:35,740 WARN  [main] migration.MigrationTool
> >>  (MigrationTool.java:530) - Following things might have
> >>  happened|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:35,746 WARN  [main] migration.MigrationTool
> >>  (MigrationTool.java:531) - 1.User might have set the time to 60
> >>  mins|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:35,747 WARN  [main] migration.MigrationTool
> >>  (MigrationTool.java:532) - 2.DMABackupInfo.inf file or CARMigraionTime
> >>  entry may not be present |<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:35,747 WARN  [main] migration.MigrationTool
> >>  (MigrationTool.java:533) - Hence setting the default migration time
> >>  |<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:35,748 INFO  [main] migration.MigrationTool
> >>  (MigrationTool.java:108) - main()-->|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:35
> >>  CDT 2012:main:connect() SQLException|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:35
> >>  CDT 2012:main:|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:35,996 WARN  [main] dbl.Connector (Connector.java:1184) -
> >>  Connector(NOT CONNECTED:
> >>  Driver=com.informix.jdbc.IfxDriver;)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) java.sql.SQLException:
> >>  Cannot open system catalog (systables).|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.a(IfxSqli.java:3457)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.E(IfxSqli.java:3774)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2580)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2496)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.executeOpenDatabase(IfxSqli.java:2021)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1622)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >>  Method)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  java.lang.reflect.Constructor.newInstance(Constructor.java:513)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxDriver.connect(IfxDriver.java:254)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  java.sql.DriverManager.getConnection(DriverManager.java:582)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  java.sql.DriverManager.getConnection(DriverManager.java:207)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.getConn(Connector.java:769)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.connect(Connector.java:677)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.connect(Connector.java:619)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:978)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:964)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.car.beans.EBSystemPreferences.loadCurrentData(EBSystemPreferences.java:171)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.car.beans.EBSystemPreferences.<init>(EBSystemPreferences.java:98)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.car.database.SchemaCreator.<clinit>(SchemaCreator.java:111)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.car.migration.MigrationTool.main(MigrationTool.java:110)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) Caused by:
> >>  java.sql.SQLException: ISAM error: Primary and Mirror chunks are
> >>  bad|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:408)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.E(IfxSqli.java:3779)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         ... 20 more|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) Thu May 17 00:33:36
> >>  CDT 2012:main:handleConnectionException: ErrorCode is
> >>  -311|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:36,007 WARN  [main] dbl.Connector (Connector.java:1187) -
> >>  handleConnectionException: ErrorCode is -311|<LVL::Debug>
> >>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:36,007 ERROR [main] beans.EBSystemPreferences
> >>  (EBSystemPreferences.java:199) - loadCurrentData(): Error while
> >>  loading Sys Pref data|<LVL::Debug>
> >>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) java.sql.SQLException:
> >>  Cannot open system catalog (systables).|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.a(IfxSqli.java:3457)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.E(IfxSqli.java:3774)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.dispatchMsg(IfxSqli.java:2580)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.receiveMessage(IfxSqli.java:2496)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.executeOpenDatabase(IfxSqli.java:2021)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqliConnect.<init>(IfxSqliConnect.java:1622)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> >>  Method)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  java.lang.reflect.Constructor.newInstance(Constructor.java:513)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxDriver.connect(IfxDriver.java:254)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  java.sql.DriverManager.getConnection(DriverManager.java:582)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  java.sql.DriverManager.getConnection(DriverManager.java:207)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.getConn(Connector.java:769)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.connect(Connector.java:677)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.connect(Connector.java:619)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:978)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.dbl.Connector.prepareStatement(Connector.java:964)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.car.beans.EBSystemPreferences.loadCurrentData(EBSystemPreferences.java:171)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.car.beans.EBSystemPreferences.<init>(EBSystemPreferences.java:98)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.car.database.SchemaCreator.<clinit>(SchemaCreator.java:111)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.cisco.ccm.car.migration.MigrationTool.main(MigrationTool.java:110)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) Caused by:
> >>  java.sql.SQLException: ISAM error: Primary and Mirror chunks are
> >>  bad|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:408)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         at
> >>  com.informix.jdbc.IfxSqli.E(IfxSqli.java:3779)|<LVL::Debug>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE)         ... 20 more|<LVL::Debug>
> >>
> >>  05/17/2012 00:33:36 component_install|(CAPTURE) 2012-05-17
> >>  00:33:36,008 ERROR [main] database.SchemaCreator
> >>  (SchemaCreator.java:128) - error while setting the manual purge status
> >>  to 0 |<LVL::Debug>
> >>  05/17/2012
> >>  _______________________________________________
> >>  cisco-voip mailing list
> >>  cisco-voip at puck.nether.net
> >>  https://puck.nether.net/mailman/listinfo/cisco-voip
> >>
> >>
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20120517/45b4a568/attachment.html>