<div dir="ltr">We had a similar issue a few months back. It turns out we exhausted our SNMP threads. <div><br></div><div>Here's some notes from the TAC case we had open:<div><br></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)"> </span><u></u><u></u></p></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">We can see that we are running into an issue with the <span class="gmail-il">SNMP</span> Master Agent on CUCM running out of threads:</span></p></div></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)"> </span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">LogLevel: APERROR</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">timestamp: 697483 (2016/07/07 14:30:41.18)</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">progname: snmpdm</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">filename: npthread_thrd.c</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">linenum: 182</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><b><span style="color:rgb(31,73,125)">message: snmpdm: new_tid: no more threads available</span></b></p></div><div><p class="MsoNormal" style="font-size:12.8px"><b><span style="color:rgb(31,73,125)"> </span></b></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">A thread will be used and released for each <span class="gmail-il">SNMP</span> request that is presented into CallManager. It will be cleared when all information has been presented in the reply. This would include the request from <span class="gmail-il">CER</span> to track devices. When attempting to query CUCM from the <span class="gmail-il">CER</span> CLI, we were getting upwards of 3 to 4 minutes of delay before a reply was being seen. This would correlate with the above snippet that were do not have available threads, as <span class="gmail-il">CER</span> will continue to retry based on the configured timers, we were able to get a response at a later time.</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)"> </span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">Action Plan:</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)"> </span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">Temporarily stop the monitoring applications that are querying into CUCM, restart the <span class="gmail-il">SNMP</span> Master Agent on the CallManager servers to clear the threads, and run through a Phone Tracking Process in <span class="gmail-il">CER</span>.</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)"> </span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">This will allow us to be sure that threads are available for <span class="gmail-il">CER</span> to retrieve the full amount of devices as expected. Phone Tracking Logs have been set to Debug, if we are not able to get the applications disabled, we will want the following logs:</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)"> </span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">-Phone Tracking Logs covering the requested Phone Tracking Process</span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)"> </span></p></div><div><p class="MsoNormal" style="font-size:12.8px"><span style="color:rgb(31,73,125)">-<span class="gmail-il">SNMP</span> Master Agent logs from the CUCM Nodes</span></p></div></blockquote><div><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p><p class="MsoNormal" style="font-size:12.8px"><u></u></p></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><font color="#444444" face="arial, helvetica, sans-serif">Joshua Lamont</font><div><font color="#444444" face="arial, helvetica, sans-serif">Senior Telecommunications Engineer</font></div><div><font color="#444444" face="arial, helvetica, sans-serif">Brown University</font></div><div><font color="#444444" face="arial, helvetica, sans-serif">office (401) 863-1003</font></div><div><font color="#444444" face="arial, helvetica, sans-serif">cell (401) 749-6913</font></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Thu, Sep 29, 2016 at 5:46 AM, Jonathan Charles <span dir="ltr"><<a href="mailto:jonvoip@gmail.com" target="_blank">jonvoip@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">So, CER is running, it sees switches, but no phone data... The tracking has a bunch of Java errors:<div><br></div><div><div><br></div><div>46633: Sep 29 05:39:10.886 UTC %CER-CER_PHONETRACKINGENGINE-<wbr>3-ERROR:EvalTask-CerIPT-03 :DiscoverySMFGetBridg.<wbr>processEntry Error in processing Cam Table Entry - java.lang.NullPointerException</div><div>46634: Sep 29 05:39:10.886 UTC %CER-CER_PHONETRACKINGENGINE-<wbr>3-ERROR:EvalTask-CerIPT-03 :DiscoverySMFGetBridg.<wbr>processEntry java.lang.NullPointerException</div><div> com.cisco.nm.ani.server.<wbr>discovery.<wbr>DiscoverySMFGetBridgeTable.<wbr>processEntry(<wbr>DiscoverySMFGetBridgeTable.<wbr>java:413)</div><div> com.cisco.nm.ani.server.<wbr>devices.C4KIOS.<wbr>C4KIOSDiscoverySMFGetBridgeTab<wbr>le.fetchFdbTable(<wbr>C4KIOSDiscoverySMFGetBridgeTab<wbr>le.java:116)</div><div> com.cisco.nm.ani.server.<wbr>discovery.<wbr>DiscoverySMFGetBridgeTable.<wbr>fetchBridgeTable(<wbr>DiscoverySMFGetBridgeTable.<wbr>java:393)</div><div> com.cisco.nm.ani.server.<wbr>discovery.<wbr>DiscoverySMFGetBridgeTable.<wbr>fetchBridgeMibTable(<wbr>DiscoverySMFGetBridgeTable.<wbr>java:113)</div><div> com.cisco.nm.ani.server.<wbr>cerIPT.<wbr>CerIPTSMFGetBridgeTable.apply(<wbr>CerIPTSMFGetBridgeTable.java:<wbr>57)</div><div> com.cisco.nm.ani.server.<wbr>framework.EvalSMFApply.apply(<wbr>EvalSMFApply.java:103)</div><div> com.cisco.nm.ani.server.<wbr>frontend.EvalPair.apply(<wbr>EvalPair.java:112)</div><div> com.cisco.nm.ani.server.<wbr>frontend.EvalTask.run(<wbr>EvalTask.java:43)</div><div> java.lang.Thread.run(Thread.<wbr>java:724)</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div>and for CCM:</div><div><br></div><div><div><br></div><div><br></div><div>46647: Sep 29 06:29:07.045 UTC %CER-CER_PHONETRACKINGENGINE-<wbr>3-ERROR:Thread-2448 :GetCCMList$1 run com.cisco.nm.lib.snmp.<wbr>futureapi.<wbr>SnmpReqTimeoutException: SnmpRequestTimeout on 10.160.22.29 while performing SnmpWalk(*) at index = -1</div><div> com.cisco.nm.lib.snmp.<wbr>futureapi.SnmpFuture.value(<wbr>SnmpFuture.java:169)</div><div> com.cisco.nm.lib.snmp.<wbr>futureapi.SnmpTableFuture.<wbr>value(SnmpTableFuture.java:<wbr>125)</div><div> com.cisco.e911.deserver.<wbr>discoveryengine.GetCCMList$1.<wbr>run(GetCCMList.java:54)</div><div> java.lang.Thread.run(Thread.<wbr>java:724)</div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>Any ideas?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div><br></div><div><br></div><div><br></div><div>Jonathan</div></font></span></div>
<br>______________________________<wbr>_________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">https://puck.nether.net/<wbr>mailman/listinfo/cisco-voip</a><br>
<br></blockquote></div><br></div></div>