[cisco-voip] FW: FW: FW: [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound

Wed Nov 17 11:22:13 EST 2010

As per Wes below, can anyone shed some light on how to work out the backtrace results:

This is what I had on one of my nodes today:

backtrace
 ===================================
 #0  0x00362a92 in strncpy () from /lib/tls/libc.so.6
#1  0x00a2cf38 in OK_BOD_strncpy () from /lib/csa/libcsa.so.6
#2  0x00a2a570 in strncpy () from /lib/csa/libcsa.so.6
#3  0x080b272b in CQBE_Parser::qbeGet (pMsgBuf_=0x14ffe000, pParsedMsg_=0x14fed5ec, pQbeString_=0x2e312e36, pNextOffset_=0x14fed46c) at qbeparser.cpp:410
#4  0x080b4e06 in CTIQBE::ProviderOpenCompletedEvent::qbeGet (this=0x80b272b, pMsgBuf_=0x14ff15c0, pNextOffset_=0x14fed46c) at qbeparser.cpp:2847
#5  0x080b94cc in CQBE_Parser::qbeParse (this=0x14ff15a0, pInputMsg_=0x14ff15c0, inputMsgLength_=89, pParsedMsg_=0x14fed5a0, parsedMsgBufferSize_=16384) at qbeparser.cpp:5558
#6  0x08095261 in CCtiInterface::receiveLoop (this=0x0) at ctireceiver.cpp:428
#7  0x00000000 in ?? ()

CHris

From: Wes Sisk [mailto:wsisk at cisco.com]
Sent: 17 November 2010 16:01
To: Lewis, Chris
Cc: Scott.Voll at wesd.org
Subject: Re: FW: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound

Hi Chris,

Please post this to cisco-voip list or the Cisco Support Community.  It's best to discuss in open forum so all can contribute and benefit.

/Wes

Lewis, Chris wrote:
Hi Scott/Wes,

How did you work out the backtrace?  I have a similar issue and was wondering how you knew it was that specific Bug?

Chris

From: cisco-voip-bounces at puck.nether.net<mailto:cisco-voip-bounces at puck.nether.net> [mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Wes Sisk
Sent: 27 January 2010 19:40
To: Voll, Scott
Cc: Cisco VoIP (cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>)
Subject: Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound

RTMT check boxes:
trace and log central
collect files
"select ccm services/applications" -> cisco extended functions, cisco extended functions report
"select system services/applications" -> cisco corefile recovery tool

/Wes

On Wednesday, January 27, 2010 2:17:24 PM, Voll, Scott <Scott.Voll at wesd.org><mailto:Scott.Voll at wesd.org> wrote:

Wes-

I feel like a dip shmoe for asking.... But how do I get the:

Core files and CEF traces?

Scott

From: Wes Sisk [mailto:wsisk at cisco.com]
Sent: Wednesday, January 27, 2010 11:00 AM
To: Voll, Scott
Cc: Cisco VoIP (cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>)
Subject: Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound

Good news: you're not the only one seeing this on 6.1.4 which is supposed to already contain the fix
bad news: no root cause yet so more investigation is required

best bet is to open a TAC case with the backtrace, version, core file, and cef traces.  it is possible the fix did not get merged properly. but that is still a discussion that needs to come out of a specific tac case.

/Wes

On Wednesday, January 27, 2010 1:42:15 PM, Voll, Scott <Scott.Voll at wesd.org><mailto:Scott.Voll at wesd.org> wrote:

6.1.4.2000.2

From: Wes Sisk [mailto:wsisk at cisco.com]
Sent: Wednesday, January 27, 2010 10:40 AM
To: Voll, Scott
Cc: Cisco VoIP (cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>)
Subject: Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound

you didn't include what version you are running ;)

Looks like
CSCso02381    CEF core due to invalid incoming QBE message header

I just updated the RNE to include the include the backtrace, check bug toolkit in 24-48 hours.

Looks like it's fixed in 6.1(3.3206.1), 6.1(4), and 7.0(1).  What version are you running?

/Wes

On Wednesday, January 27, 2010 12:14:32 PM, Voll, Scott <Scott.Voll at wesd.org><mailto:Scott.Voll at wesd.org> wrote:

Where do I find this "backtrace" in the bug toolkit?

Here is my backtrace:

#0  0x0032ca92 in strncpy () from /lib/tls/libc.so.6
#1  0x00e17f38 in OK_BOD_strncpy () from /lib/csa/libcsa.so.6
#2  0x00e15570 in strncpy () from /lib/csa/libcsa.so.6
#3  0x080b26eb in CQBE_Parser::qbeGet (pMsgBuf_=0x1756f000, pParsedMsg_=0x1755e5ec, pQbeString_=0x2e312e36, pNextOffset_=0x1755e46c) at qbeparser.cpp:410
#4  0x080b4dc6 in CTIQBE::ProviderOpenCompletedEvent::qbeGet (this=0x80b26eb, pMsgBuf_=0x175625c0, pNextOffset_=0x1755e46c) at qbeparser.cpp:2847
#5  0x080b948c in CQBE_Parser::qbeParse (this=0x175625a0, pInputMsg_=0x175625c0, inputMsgLength_=89, pParsedMsg_=0x1755e5a0, parsedMsgBufferSize_=16384) at qbeparser.cpp:5542
#6  0x08095221 in CCtiInterface::receiveLoop (this=0x0) at ctireceiver.cpp:428
#7  0x00000000 in ?? ()

Scott

From: Wes Sisk [mailto:wsisk at cisco.com]
Sent: Wednesday, January 27, 2010 8:46 AM
To: Voll, Scott
Cc: Cisco VoIP (cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>)
Subject: Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound

yep, it means your cef process generated a core dump.  i.e. it crashed.  this is the equivalent of windows dr.watson and user.dmp.

First level diagnosis:
from the CLI use 'utils core list' to identify the core file and then 'utils core analyze' to generate the backtrace from the core file.  you *should* be able to search for the backtrace in bug toolkit to find the relevant defect. if btk is not cooperating then open a tac case and provide your CM version and the backtrace.  That is adequate to identify the defect for known crashes.

Second level diagnosis:
If the backtrace does not match a known signature then we have to diagnose the core.  That requires:
- the actual cef core file (core files may be very large)
- the backtrace from the core file
- cef traces for 1 hour leading up to the failure
- possibly other traces such as cm SDI/SDL for the same timeframe

/Wes

On Wednesday, January 27, 2010 10:41:41 AM, Voll, Scott <Scott.Voll at wesd.org><mailto:Scott.Voll at wesd.org> wrote:

I'm not finding much on a Google search.  Any idea what this means?

I'm guessing a Cisco Extended Function crashed on my first node, but not quite sure what needs to be done or if this is a tac case.

>From a TAC point of view which RTMT logs should I be getting if I need to open a ticket?

This is the first time I've seen this error.

Thanks

Scott

PS. CM 6.1.4-2000.2

From: RTMT_Admin at wesd.org<mailto:RTMT_Admin at wesd.org> [mailto:RTMT_Admin at wesd.org]
Sent: Wednesday, January 27, 2010 6:06 AM
To: Voll, Scott
Subject: [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound

CoreDumpFileFound TotalCoresFound : 1 CoreDetails : The following lists up to 6 cores dumped by corresponding applications. Core1 : Cisco Extended Functions (core.13530.11.cef.1264601139) AppID : Cisco Log Partition Monitoring Tool ClusterID : NodeID : CM1 . The alarm is generated on Wed Jan 27 06:05:44 PST 2010.

________________________________

_______________________________________________

cisco-voip mailing list

cisco-voip at puck.nether.net<mailto:cisco-voip at puck.nether.net>

https://puck.nether.net/mailman/listinfo/cisco-voip

The information contained in this message and its attachments

is intended only for the private and confidential use of the

intended recipient(s).  If you are not the intended recipient

(or have received this e-mail in error) please notify the

sender immediately and destroy this e-mail. Any unauthorized

copying, disclosure or distribution of the material in this e-

mail is strictly prohibited.

The information contained in this message and its attachments 
is intended only for the private and confidential use of the 
intended recipient(s).  If you are not the intended recipient 
(or have received this e-mail in error) please notify the 
sender immediately and destroy this e-mail. Any unauthorized 
copying, disclosure or distribution of the material in this e-
mail is strictly prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20101117/5cf93df3/attachment.html>