[cisco-voip] CER 11...

[Lamont, Joshua]

We had a similar issue a few months back. It turns out we exhausted our
SNMP threads.

Here's some notes from the TAC case we had open:



We can see that we are running into an issue with the SNMP Master Agent on
CUCM running out of threads:



LogLevel:    APERROR

timestamp:   697483 (2016/07/07 14:30:41.18)

progname:    snmpdm

filename:    npthread_thrd.c

linenum:     182

*message:     snmpdm: new_tid: no more threads available*



A thread will be used and released for each SNMP request that is presented
into CallManager. It will be cleared when all information has been
presented in the reply. This would include the request from CER to track
devices. When attempting to query CUCM from the CER CLI, we were getting
upwards of 3 to 4 minutes of delay before a reply was being seen. This
would correlate with the above snippet that were do not have available
threads, as CER will continue to retry based on the configured timers, we
were able to get a response at a later time.



Action Plan:



Temporarily stop the monitoring applications that are querying into CUCM,
restart the SNMP Master Agent on the CallManager servers to clear the
threads, and run through a Phone Tracking Process in CER.



This will allow us to be sure that threads are available for CER to
retrieve the full amount of devices as expected. Phone Tracking Logs have
been set to Debug, if we are not able to get the applications disabled, we
will want the following logs:



-Phone Tracking Logs covering the requested Phone Tracking Process



-SNMP Master Agent logs from the CUCM Nodes


Joshua Lamont
Senior Telecommunications Engineer
Brown University
office (401) 863-1003
cell    (401) 749-6913

On Thu, Sep 29, 2016 at 5:46 AM, Jonathan Charles <jonvoip at gmail.com> wrote:

> So, CER is running, it sees switches, but no phone data... The tracking
> has a bunch of Java errors:
>
>
> 46633: Sep 29 05:39:10.886 UTC %CER-CER_PHONETRACKINGENGINE-3-ERROR:EvalTask-CerIPT-03
>    :DiscoverySMFGetBridg.processEntry         Error in processing Cam
> Table Entry - java.lang.NullPointerException
> 46634: Sep 29 05:39:10.886 UTC %CER-CER_PHONETRACKINGENGINE-3-ERROR:EvalTask-CerIPT-03
>    :DiscoverySMFGetBridg.processEntry
> java.lang.NullPointerException
>
> com.cisco.nm.ani.server.discovery.DiscoverySMFGetBridgeTable.processEntry(
> DiscoverySMFGetBridgeTable.java:413)
>
> com.cisco.nm.ani.server.devices.C4KIOS.C4KIOSDiscoverySMFGetBridgeTab
> le.fetchFdbTable(C4KIOSDiscoverySMFGetBridgeTable.java:116)
>
> com.cisco.nm.ani.server.discovery.DiscoverySMFGetBridgeTable.
> fetchBridgeTable(DiscoverySMFGetBridgeTable.java:393)
>
> com.cisco.nm.ani.server.discovery.DiscoverySMFGetBridgeTable.
> fetchBridgeMibTable(DiscoverySMFGetBridgeTable.java:113)
>
> com.cisco.nm.ani.server.cerIPT.CerIPTSMFGetBridgeTable.apply(
> CerIPTSMFGetBridgeTable.java:57)
>
> com.cisco.nm.ani.server.framework.EvalSMFApply.apply(
> EvalSMFApply.java:103)
>
> com.cisco.nm.ani.server.frontend.EvalPair.apply(EvalPair.java:112)
>
> com.cisco.nm.ani.server.frontend.EvalTask.run(EvalTask.java:43)
>
> java.lang.Thread.run(Thread.java:724)
>
>
>
>
> and for CCM:
>
>
>
> 46647: Sep 29 06:29:07.045 UTC %CER-CER_PHONETRACKINGENGINE-3-ERROR:Thread-2448
>           :GetCCMList$1         run                  com.cisco.nm.lib.snmp.
> futureapi.SnmpReqTimeoutException: SnmpRequestTimeout on 10.160.22.29
> while performing SnmpWalk(*) at index = -1
>
> com.cisco.nm.lib.snmp.futureapi.SnmpFuture.value(SnmpFuture.java:169)
>
> com.cisco.nm.lib.snmp.futureapi.SnmpTableFuture.
> value(SnmpTableFuture.java:125)
>
> com.cisco.e911.deserver.discoveryengine.GetCCMList$1.
> run(GetCCMList.java:54)
>
> java.lang.Thread.run(Thread.java:724)
>
>
>
>
>
> Any ideas?
>
>
>
>
> Jonathan
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20160929/63c3e6e3/attachment.html>