[cisco-voip] FW: FW: FW: [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound

Wes Sisk wsisk at cisco.com
Wed Nov 17 14:47:54 EST 2010 

What version are you on?

Core dumps are read from bottom to top.

receiveLoop..qbeGet exactly matches the backtrace in the Release-note of 
CSCso02381 that is visible in Cisco Bug Toolkit.  That looks like a good 
starting point.

/Wes

Lewis, Chris wrote:
>
> As per Wes below, can anyone shed some light on how to work out the 
> backtrace results:
>
>  
>
> This is what I had on one of my nodes today:
>
>  
>
> backtrace
>
>  ===================================
>
>  #0  0x00362a92 in strncpy () from /lib/tls/libc.so.6
>
> #1  0x00a2cf38 in OK_BOD_strncpy () from /lib/csa/libcsa.so.6
>
> #2  0x00a2a570 in strncpy () from /lib/csa/libcsa.so.6
>
> #3  0x080b272b in CQBE_Parser::qbeGet (pMsgBuf_=0x14ffe000, 
> pParsedMsg_=0x14fed5ec, pQbeString_=0x2e312e36, 
> pNextOffset_=0x14fed46c) at qbeparser.cpp:410
>
> #4  0x080b4e06 in CTIQBE::ProviderOpenCompletedEvent::qbeGet 
> (this=0x80b272b, pMsgBuf_=0x14ff15c0, pNextOffset_=0x14fed46c) at 
> qbeparser.cpp:2847
>
> #5  0x080b94cc in CQBE_Parser::qbeParse (this=0x14ff15a0, 
> pInputMsg_=0x14ff15c0, inputMsgLength_=89, pParsedMsg_=0x14fed5a0, 
> parsedMsgBufferSize_=16384) at qbeparser.cpp:5558
>
> #6  0x08095261 in CCtiInterface::receiveLoop (this=0x0) at 
> ctireceiver.cpp:428
>
> #7  0x00000000 in ?? ()
>
>  
>
> CHris
>
>  
>
> *From:* Wes Sisk [mailto:wsisk at cisco.com]
> *Sent:* 17 November 2010 16:01
> *To:* Lewis, Chris
> *Cc:* Scott.Voll at wesd.org
> *Subject:* Re: FW: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> Hi Chris,
>
> Please post this to cisco-voip list or the Cisco Support Community.  
> It's best to discuss in open forum so all can contribute and benefit.
>
> /Wes
>
> Lewis, Chris wrote:
>
> Hi Scott/Wes,
>
>  
>
> How did you work out the backtrace?  I have a similar issue and was 
> wondering how you knew it was that specific Bug?
>
>  
>
> Chris
>
>  
>
> *From:* cisco-voip-bounces at puck.nether.net 
> <mailto:cisco-voip-bounces at puck.nether.net> 
> [mailto:cisco-voip-bounces at puck.nether.net] *On Behalf Of *Wes Sisk
> *Sent:* 27 January 2010 19:40
> *To:* Voll, Scott
> *Cc:* Cisco VoIP (cisco-voip at puck.nether.net 
> <mailto:cisco-voip at puck.nether.net>)
> *Subject:* Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> RTMT check boxes:
> trace and log central
> collect files
> "select ccm services/applications" -> cisco extended functions, cisco 
> extended functions report
> "select system services/applications" -> cisco corefile recovery tool
>
> /Wes
>
> On Wednesday, January 27, 2010 2:17:24 PM, Voll, Scott 
> <Scott.Voll at wesd.org> <mailto:Scott.Voll at wesd.org> wrote:
>
> Wes---
>
>  
>
> I feel like a dip shmoe for asking.... But how do I get the:
>
>  
>
> Core files and CEF traces?
>
>  
>
> Scott
>
>  
>
> *From:* Wes Sisk [mailto:wsisk at cisco.com]
> *Sent:* Wednesday, January 27, 2010 11:00 AM
> *To:* Voll, Scott
> *Cc:* Cisco VoIP (cisco-voip at puck.nether.net 
> <mailto:cisco-voip at puck.nether.net>)
> *Subject:* Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> Good news: you're not the only one seeing this on 6.1.4 which is 
> supposed to already contain the fix
> bad news: no root cause yet so more investigation is required
>
> best bet is to open a TAC case with the backtrace, version, core file, 
> and cef traces.  it is possible the fix did not get merged properly. 
> but that is still a discussion that needs to come out of a specific 
> tac case.
>
> /Wes
>
> On Wednesday, January 27, 2010 1:42:15 PM, Voll, Scott 
> <Scott.Voll at wesd.org> <mailto:Scott.Voll at wesd.org> wrote:
>
>
> 6.1.4.2000.2
>
>  
>
> *From:* Wes Sisk [mailto:wsisk at cisco.com]
> *Sent:* Wednesday, January 27, 2010 10:40 AM
> *To:* Voll, Scott
> *Cc:* Cisco VoIP (cisco-voip at puck.nether.net 
> <mailto:cisco-voip at puck.nether.net>)
> *Subject:* Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> you didn't include what version you are running ;)
>
> Looks like
> CSCso02381    CEF core due to invalid incoming QBE message header
>
> I just updated the RNE to include the include the backtrace, check bug 
> toolkit in 24-48 hours.
>
> Looks like it's fixed in 6.1(3.3206.1), 6.1(4), and 7.0(1).  What 
> version are you running?
>
> /Wes
>
> On Wednesday, January 27, 2010 12:14:32 PM, Voll, Scott 
> <Scott.Voll at wesd.org> <mailto:Scott.Voll at wesd.org> wrote:
>
>
>
> Where do I find this "backtrace" in the bug toolkit?
>
>  
>
> Here is my backtrace:
>
>  
>
> #0  0x0032ca92 in strncpy () from /lib/tls/libc.so.6
>
> #1  0x00e17f38 in OK_BOD_strncpy () from /lib/csa/libcsa.so.6
>
> #2  0x00e15570 in strncpy () from /lib/csa/libcsa.so.6
>
> #3  0x080b26eb in CQBE_Parser::qbeGet (pMsgBuf_=0x1756f000, 
> pParsedMsg_=0x1755e5ec, pQbeString_=0x2e312e36, 
> pNextOffset_=0x1755e46c) at qbeparser.cpp:410
>
> #4  0x080b4dc6 in CTIQBE::ProviderOpenCompletedEvent::qbeGet 
> (this=0x80b26eb, pMsgBuf_=0x175625c0, pNextOffset_=0x1755e46c) at 
> qbeparser.cpp:2847
>
> #5  0x080b948c in CQBE_Parser::qbeParse (this=0x175625a0, 
> pInputMsg_=0x175625c0, inputMsgLength_=89, pParsedMsg_=0x1755e5a0, 
> parsedMsgBufferSize_=16384) at qbeparser.cpp:5542
>
> #6  0x08095221 in CCtiInterface::receiveLoop (this=0x0) at 
> ctireceiver.cpp:428
>
> #7  0x00000000 in ?? ()
>
>  
>
> Scott
>
>  
>
> *From:* Wes Sisk [mailto:wsisk at cisco.com]
> *Sent:* Wednesday, January 27, 2010 8:46 AM
> *To:* Voll, Scott
> *Cc:* Cisco VoIP (cisco-voip at puck.nether.net 
> <mailto:cisco-voip at puck.nether.net>)
> *Subject:* Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> yep, it means your cef process generated a core dump.  i.e. it 
> crashed.  this is the equivalent of windows dr.watson and user.dmp.
>
> First level diagnosis:
> from the CLI use 'utils core list' to identify the core file and then 
> 'utils core analyze' to generate the backtrace from the core file.  
> you *should* be able to search for the backtrace in bug toolkit to 
> find the relevant defect. if btk is not cooperating then open a tac 
> case and provide your CM version and the backtrace.  That is adequate 
> to identify the defect for known crashes.
>
> Second level diagnosis:
> If the backtrace does not match a known signature then we have to 
> diagnose the core.  That requires:
> - the actual cef core file (core files may be very large)
> - the backtrace from the core file
> - cef traces for 1 hour leading up to the failure
> - possibly other traces such as cm SDI/SDL for the same timeframe
>
> /Wes
>
>
> On Wednesday, January 27, 2010 10:41:41 AM, Voll, Scott 
> <Scott.Voll at wesd.org> <mailto:Scott.Voll at wesd.org> wrote:
>
>
>
>
> I'm not finding much on a Google search.  Any idea what this means?
>
>  
>
> I'm guessing a Cisco Extended Function crashed on my first node, but 
> not quite sure what needs to be done or if this is a tac case.
>
>  
>
> From a TAC point of view which RTMT logs should I be getting if I need 
> to open a ticket?
>
>  
>
> This is the first time I've seen this error.
>
>  
>
> Thanks
>
>  
>
> Scott
>
>  
>
> PS. CM 6.1.4-2000.2
>
>  
>
> *From:* RTMT_Admin at wesd.org <mailto:RTMT_Admin at wesd.org> 
> [mailto:RTMT_Admin at wesd.org]
> *Sent:* Wednesday, January 27, 2010 6:06 AM
> *To:* Voll, Scott
> *Subject:* [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound
>
>  
>
> CoreDumpFileFound TotalCoresFound : 1 CoreDetails : The following 
> lists up to 6 cores dumped by corresponding applications. Core1 : 
> Cisco Extended Functions (core.13530.11.cef.1264601139) AppID : Cisco 
> Log Partition Monitoring Tool ClusterID : NodeID : CM1 . The alarm is 
> generated on Wed Jan 27 06:05:44 PST 2010.
>
>  
>  
>  
>  
>  
>  
>
>
> ------------------------------------------------------------------------
>
>
>  
>  
>  
>  
>   
>  
>  
>   
>  
>  
>   
>  
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net <mailto:cisco-voip at puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-voip
>   
>
>  
>
>  
>
>  
>
>  
>
> The information contained in this message and its attachments 
> is intended only for the private and confidential use of the 
> intended recipient(s).  If you are not the intended recipient 
> (or have received this e-mail in error) please notify the 
> sender immediately and destroy this e-mail. Any unauthorized 
> copying, disclosure or distribution of the material in this e-
> mail is strictly prohibited.
>   
> The information contained in this message and its attachments 
> is intended only for the private and confidential use of the 
> intended recipient(s).  If you are not the intended recipient 
> (or have received this e-mail in error) please notify the 
> sender immediately and destroy this e-mail. Any unauthorized 
> copying, disclosure or distribution of the material in this e-
> mail is strictly prohibited.
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20101117/8b695a41/attachment.html>

More information about the cisco-voip mailing list