<div dir="ltr">Did you download Perfmon logs? Might just be high memory in general causing the issue. I'd use the Perfmon logs to check what processes were using up all the memory at that time.</div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jan 22, 2016 at 10:45 AM, Erick Bergquist <span dir="ltr"><<a href="mailto:erickbee@gmail.com" target="_blank">erickbee@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">This happened on CUCM 10.5.2.10000-7 and I'm not finding a bug for<br>
this backtrace info in bug toolkit or the 10.5.2 SU1, SU2, SU3 release<br>
notes.<br>
<br>
<br>
>From the core analysis,<br>
<br>
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols<br>
found)...done.<br>
Loaded symbols for /lib/libnss_files.so.2<br>
Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done.<br>
Loaded symbols for /lib/libnss_dns.so.2<br>
Core was generated by `/usr/local/cm/bin/ccm'.<br>
Program terminated with signal 11, Segmentation fault.<br>
#0 0xf6a28b55 in _int_malloc () from /lib/libc.so.6<br>
<br>
====================================<br>
backtrace - CUCM<br>
===================================<br>
#0 0xf6a28b55 in _int_malloc () from /lib/libc.so.6<br>
#1 0xf6a29d9e in malloc () from /lib/libc.so.6<br>
#2 0x09be122c in AlternateSyslogAlarmMonitor::manageLogs<br>
(this=0xd7d74f8, severity=6, strBuf=<value optimized out>) at<br>
AlternateSyslogAlarmMonitor.cpp:162<br>
#3 0x09be13f6 in AlternateSyslogAlarmMonitor::AlarmThreadProc<br>
(pParameter=0xd7d74f8) at AlternateSyslogAlarmMonitor.cpp:61<br>
#4 0xf6d005a8 in ACE_OS_Thread_Adapter::invoke (this=0xd7d8dc8) at<br>
OS_Thread_Adapter.cpp:103<br>
#5 0xf6cc05f1 in ace_thread_adapter (args=0xd7d8dc8) at<br>
Base_Thread_Adapter.cpp:126<br>
#6 0xf6996b39 in start_thread () from /lib/libpthread.so.0<br>
#7 0xf6a97c1e in clone () from /lib/libc.so.6<br>
====================================<br>
info registers<br>
====================================<br>
eax 0x198408<br>
ecx 0xf6b483d0-155941936<br>
edx 0xf6b483a0-155941984<br>
ebx 0xf6b46ff4-155947020<br>
esp 0xf5d0ab100xf5d0ab10<br>
ebp 0xf5d0ab980xf5d0ab98<br>
esi 0xf4bc0b8256622776<br>
edi 0x00<br>
eip 0xf6a28b550xf6a28b55 <_int_malloc+725><br>
eflags 0x10212[ AF IF RF ]<br>
cs 0x2335<br>
ss 0x2b43<br>
ds 0x2b43<br>
es 0x2b43<br>
fs 0x00<br>
gs 0x6399<br>
====================================<br>
info threads<br>
====================================<br>
47 Thread 0xe750bb70 (LWP 30608) 0xf6a237f4 in _IO_flush_all_lockp<br>
() from /lib/libc.so.6<br>
46 Thread 0xf5effb70 (LWP 29077) 0xf77d0430 in __kernel_vsyscall ()<br>
45 Thread 0xf63ffb70 (LWP 29052) 0xf77d0430 in __kernel_vsyscall ()<br>
44 Thread 0xf67d0b70 (LWP 29048) 0xf77d0430 in __kernel_vsyscall ()<br>
43 Thread 0xf67e0710 (LWP 25984) 0xf77d0430 in __kernel_vsyscall ()<br>
42 Thread 0xe6f2fb70 (LWP 30611) 0xf77d0430 in __kernel_vsyscall ()<br>
<br>
<br>
No errors/etc in the Syslog at the time other then errors on the call<br>
manager service failing.<br>
<br>
However, the SDL CCM traces show of a timer expiring and appears it<br>
shuts down services due to the timer issue.<br>
<br>
>From the following cisco doc,<br>
<a href="http://www.cisco.com/c/en/us/support/docs/voice-unified-communications/unified-communications-manager-callmanager/46806-cm-crashes-and-shutdowns.html" rel="noreferrer" target="_blank">http://www.cisco.com/c/en/us/support/docs/voice-unified-communications/unified-communications-manager-callmanager/46806-cm-crashes-and-shutdowns.html</a><br>
it looks like the SDL timer thread stopped and service shutdown?<br>
This doc is from CCM 3.x and 4.x time frame. Are the timers mentioned<br>
in this doc still true for 10.5 - where it shuts down at 16 seconds<br>
for SDL timer thread or 20 seconds for SDL router thread.<br>
<br>
<br>
09465137.000 |23:55:15.416 |Stopping |<br>
| |SdlTCPConnection(1,100,14,37)<br>
|SdlTCPConnection(1,100,14,37) |<br>
|NumOfCurrentInstances: 39<br>
<br>
09465138.000 |23:55:15.448 |SdlSig |SdlVerificationTimer<br>
|monitoring |CMProcMon(1,100,234,1)<br>
|SdlTimerService(1,100,3,1) |1,100,234,1.3139616^*^*<br>
|*TraceFlagOverrode<br>
<br>
09465138.001 |23:55:15.448 |AppInfo |CMProcMon::star_sdlVerification<br>
- Test Timer exceeded minimum timer latency threshold of 1000<br>
milliseconds, Actual latency: 54360 milliseconds<br>
<br>
09465139.000 |23:55:16.135 |AppInfo |CallManagerFailure - Indicates<br>
an internal failure in Unified CM Host Name of Hosting Node:CUCM1 Host<br>
Node IP Address:192.168.1.5 Reason code:4 Additional Text: App<br>
ID:Cisco CallManager Cluster ID:StandAloneCluster Node ID:CUCM1<br>
<br>
09465140.000 |23:55:16.135 |AlarmErr |AlarmClass: CallManager,<br>
AlarmName: CallManagerFailure, AlarmSeverity: Critical, AlarmMessage:<br>
, AlarmDescription: Indicates an internal failure in Unified CM,<br>
AlarmParameters: HostName:CUCM1, IPAddress:192.168.1.5, Reason:4,<br>
Text:, AppID:Cisco CallManager, ClusterID:StandAloneCluster,<br>
NodeID:CUCM1<br>
<br>
09465141.000 |23:55:17.465 |AppInfo |CallManagerFailure - Indicates<br>
an internal failure in Unified CM Host Name of Hosting Node:CUCM1 Host<br>
Node IP Address:192.168.1.5 Reason code:4 Additional Text:CCM<br>
Intentional Abort: SignalName: SdlVerificationTimer, DestPID:<br>
CMProcMon[1:100:234:1] App ID:Cisco CallManager Cluster<br>
ID:StandAloneCluster Node ID:CUCM1<br>
<br>
09465142.000 |23:55:17.465 |AlarmErr |AlarmClass: CallManager,<br>
AlarmName: CallManagerFailure, AlarmSeverity: Critical, AlarmMessage:<br>
, AlarmDescription: Indicates an internal failure in Unified CM,<br>
AlarmParameters: HostName:CUCM1, IPAddress:192.168.1.5, Reason:4,<br>
Text:CCM Intentional Abort: SignalName: SdlVerificationTimer, DestPID:<br>
CMProcMon[1:100:234:1], AppID:Cisco CallManager,<br>
ClusterID:StandAloneCluster, NodeID:CUCM1,<br>
<br>
09465138.002 |23:55:17.566 |AppInfo |TimerThreadSlowed - Timer thread<br>
has slowed beyond acceptable limits App ID:Cisco CallManager Cluster<br>
ID:StandAloneCluster Node ID:CUCM1<br>
<br>
09465138.003 |23:55:17.566 |AlarmErr |AlarmClass: CallManager,<br>
AlarmName: TimerThreadSlowed, AlarmSeverity: Critical, AlarmMessage: ,<br>
AlarmDescription: Timer thread has slowed beyond acceptable limits,<br>
AlarmParameters: AppID:Cisco CallManager,<br>
ClusterID:StandAloneCluster, NodeID:CUCM1,<br>
<br>
09465143.000 |23:55:17.566 |SdlSig |ReapOldTokenRegistrationsTimer<br>
|wait |SIPStationInit(1,100,72,1)<br>
|SdlTimerService(1,100,3,1) |1,100,146,1.1^*^*<br>
|[R:H-H:36,N:55,L:1,V:1,Z:0,D:0]<br>
_______________________________________________<br>
cisco-voip mailing list<br>
<a href="mailto:cisco-voip@puck.nether.net">cisco-voip@puck.nether.net</a><br>
<a href="https://puck.nether.net/mailman/listinfo/cisco-voip" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-voip</a><br>
</blockquote></div><br></div>