<div dir="ltr">Run "show network cluster" to make sure both nodes are authenticated/connected.<div><br></div><div>Then run "utils dbreplication stop" on pub and sub, run "utils dbreplication dropadmindb" on pub and sub, then run "utils dbreplication reset all" from the publisher once everything else has completed.</div><div><br></div><div>You can run "file list activelog cm/trace/dbl detail date" on the publisher to watch the files. You should see the CDR defines happen immediately. You can use "file tail activelog cm/trace/dbl/filename" to watch the files in real-time. Your CDR define file should show the result of "64" if it was successful. Then just wait 5 minutes (or however long your repltimeout is set to) and you should see the broadcast go out from the publisher.</div><div><br></div><div>Then just sit back and wait/monitor the broadcast via tail if you want.</div><div><br></div><div>No harm in resetting replication during the day. Just never ever use the clusterreset command and you'll be fine.</div><div><br></div><div>Brian</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 16, 2015 at 8:56 AM, Martin Schmuker <span dir="ltr"><<a href="mailto:ms@bilobit.com" target="_blank">ms@bilobit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="DE" link="blue" vlink="purple">
<div>
<p class="MsoNormal">All,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span lang="EN-US">I have problems with a CUCM cluster at one customer. He is running 8.6.2.20000-2.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Cluster was running fine in the past. Some months ago second machine lost power (UPS problems) and since this time sometimes we have problems that cluster is reporting as not working. After a reboot of both machines they
are running fine. <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Today publisher was stuck on 100% CPU and all phones were in “registering” state. But subscriber did not register any phones. After rebooting both machines again everything is fine, but I’m not happy with this situation.
Can you see in the following output if cluster is running fine?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">When I issue </span><span lang="EN-US" style="font-family:"Courier New"">utils dbreplication runtimestate</span><span lang="EN-US"> on
<b>publisher (=cucm01)</b> I get:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">admin:utils dbreplication runtimestate<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">DB and Replication Services: ALL RUNNING<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Cluster Replication State: Replication status command started at: 2015-02-16-14-32<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> Replication status command COMPLETED 541 tables checked out of 541<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> No Errors or Mismatches found.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> Use 'file view activelog cm/trace/dbl/sdi/ReplicationStatus.2015_02_16_14_32_48.out' to see the details<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">DB Version: ccm8_6_2_20000_2<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Number of replicated tables: 541<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Cluster Detailed View from PUB (2 Servers):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> PING REPLICATION REPL. DBver& REPL. REPLICATION SETUP<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">SERVER-NAME IP ADDRESS (msec) RPC? STATUS QUEUE TABLES LOOP? (RTMT) & details<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">----------- ------------ ------ ---- ----------- ----- ------- ----- -----------------<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">s-lx-cucm01 192.168.1.10 0.082 Yes Connected 0 match Yes (2) PUB Setup Completed<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">cucm02 192.168.1.11 0.244 Yes Connected 0 match N/A (0) Setup Requested<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US">On Subscriber (=cucm02):<u></u><u></u></span></b></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">admin:utils dbreplication runtimestate<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">DB and Replication Services: ALL RUNNING<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Cluster Replication State: Only available on the PUB<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">DB Version: ccm8_6_2_20000_2<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Number of replicated tables: 541<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Cluster Detailed View from SUB (2 Servers):<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> PING REPLICATION REPL. DBver& REPL. REPLICATION SETUP<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">SERVER-NAME IP ADDRESS (msec) RPC? STATUS QUEUE TABLES LOOP? (RTMT)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">----------- ------------ ------ ---- ----------- ----- ------- ----- -----------------<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">s-lx-cucm01 192.168.1.10 0.212 Yes Connected 0 match No (2)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">cucm02 192.168.1.11 0.055 Yes Connected 0 match No (0)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Output of <b>utils dbreplication status on pub</b>:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">admin:file view activelog cm/trace/dbl/sdi/ReplicationStatus.2015_02_16_14_32_48.out<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">SERVER ID STATE STATUS QUEUE CONNECTION CHANGED<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">-----------------------------------------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">g_cucm02_ccm8_6_2_20000_2 3 Active Connected 0 Feb 16 14:30:50<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">g_s_lx_cucm01_ccm8_6_2_20000_2 2 Active Local 0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">-------------------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">No Errors or Mismatches found.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Replication status is good on all available servers.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">utils dbreplication status output<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">To determine if replication is suspect, look for the following:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> (1) Number of rows in a table do not match on all nodes.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> (2) Non-zero values occur in any of the other output columns for a table<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Sync target server is not defined for the replicate ccmdbtemplate_s_lx_cucm01_ccm8_6_2_20000_2_1_118_typedberrors<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Feb 16 2015 14:32:53 ------ Table scan for ccmdbtemplate_s_lx_cucm01_ccm8_6_2_20000_2_1_118_typedberrors end ---------<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">And on sub:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">admin:file view activelog cm/trace/dbl/sdi/ReplicationStatus.2015_02_16_14_34_37.out<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">SERVER ID STATE STATUS QUEUE CONNECTION CHANGED<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">-----------------------------------------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">g_cucm02_ccm8_6_2_20000_2 3 Active Local 0<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">g_s_lx_cucm01_ccm8_6_2_20000_2 2 Active Connected 0 Feb 16 14:30:49<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">-------------------------------------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">No Errors or Mismatches found.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Replication status is good on all available servers.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">utils dbreplication status output<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">To determine if replication is suspect, look for the following:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> (1) Number of rows in a table do not match on all nodes.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> (2) Non-zero values occur in any of the other output columns for a table<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Sync target server is not defined for the replicate ccmdbtemplate_s_lx_cucm01_ccm8_6_2_20000_2_1_118_typedberrors<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">Feb 16 2015 14:34:43 ------ Table scan for ccmdbtemplate_s_lx_cucm01_ccm8_6_2_20000_2_1_118_typedberrors end ---------<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">show perf query class "Number of Replicates Created and State of Replication" on publisher shows:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">==>query class :<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New"">- Perf class (Number of Replicates Created and State of Replication) has instances and values:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> ReplicateCount -> Number of Replicates Created = 541<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Courier New""> ReplicateCount -> Replicate_State = 2<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">So a replicate state of “2” means everything ok, correct?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">What can I do? Is it safe to run dbreplication reset all?
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks for your ideas, Martin<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
</div>
<br>_______________________________________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
<br></blockquote></div><br></div>