[cisco-voip] FW: FW: FW: [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound

Wes Sisk wsisk at cisco.com
Thu Nov 18 08:20:57 EST 2010


Hi Chris,

In general all crashes are a result of a defect.  There are a few 
exceptions such as intentional aborts (code red, cpu starvation, etc..).

Given that we know to search for product defects.  Cisco provide 
BugToolkit for that purpose.  BugToolkit displays the defect headline 
and Release-note.  The information provided there is of varying quality, 
but it is a starting point.

Then it's just semantics:
1. login to bug toolkit
2. select your product (communications manager)
3. select your version or leave it as ANY
4. enter your keyword search - CQBE_Parser::qbeGet is the last "cisco" 
function. functions after that (libcs*) are linux core libraries.
5. click search and wait. BugToolkit isn't the fastest.  But in this 
instance it does successfully find CSCso02381.

This is a typical fist step for troubleshooting any crash.  If the 
output is fruitful then you're done.  You may change the search criteria 
some, such as version string or check similar products.  After that the 
next step would be careful inspection of any traces produced by the 
process leading up to the crash.

/Wes

Lewis, Chris wrote:
>
> Thanks Wes.
>
>  
>
> I am on 6.1.4.2190-3
>
>  
>
> How did you search for the BugID  from that?
>
>  
>
> If you don't know the Bug exists in the first place how do you know 
> where to search? 
>
>  
>
> Chris
>
>  
>
> *From:* Wes Sisk [mailto:wsisk at cisco.com]
> *Sent:* 17 November 2010 19:48
> *To:* Lewis, Chris
> *Cc:* cisco-voip at puck.nether.net
> *Subject:* Re: [cisco-voip] FW: FW: FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> What version are you on?
>
> Core dumps are read from bottom to top.
>
> receiveLoop..qbeGet exactly matches the backtrace in the Release-note 
> of CSCso02381 that is visible in Cisco Bug Toolkit.  That looks like a 
> good starting point.
>
> /Wes
>
> Lewis, Chris wrote:
>
> As per Wes below, can anyone shed some light on how to work out the 
> backtrace results:
>
>  
>
> This is what I had on one of my nodes today:
>
>  
>
> backtrace
>
>  ===================================
>
>  #0  0x00362a92 in strncpy () from /lib/tls/libc.so.6
>
> #1  0x00a2cf38 in OK_BOD_strncpy () from /lib/csa/libcsa.so.6
>
> #2  0x00a2a570 in strncpy () from /lib/csa/libcsa.so.6
>
> #3  0x080b272b in CQBE_Parser::qbeGet (pMsgBuf_=0x14ffe000, 
> pParsedMsg_=0x14fed5ec, pQbeString_=0x2e312e36, 
> pNextOffset_=0x14fed46c) at qbeparser.cpp:410
>
> #4  0x080b4e06 in CTIQBE::ProviderOpenCompletedEvent::qbeGet 
> (this=0x80b272b, pMsgBuf_=0x14ff15c0, pNextOffset_=0x14fed46c) at 
> qbeparser.cpp:2847
>
> #5  0x080b94cc in CQBE_Parser::qbeParse (this=0x14ff15a0, 
> pInputMsg_=0x14ff15c0, inputMsgLength_=89, pParsedMsg_=0x14fed5a0, 
> parsedMsgBufferSize_=16384) at qbeparser.cpp:5558
>
> #6  0x08095261 in CCtiInterface::receiveLoop (this=0x0) at 
> ctireceiver.cpp:428
>
> #7  0x00000000 in ?? ()
>
>  
>
> CHris
>
>  
>
> *From:* Wes Sisk [mailto:wsisk at cisco.com]
> *Sent:* 17 November 2010 16:01
> *To:* Lewis, Chris
> *Cc:* Scott.Voll at wesd.org <mailto:Scott.Voll at wesd.org>
> *Subject:* Re: FW: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> Hi Chris,
>
> Please post this to cisco-voip list or the Cisco Support Community.  
> It's best to discuss in open forum so all can contribute and benefit.
>
> /Wes
>
> Lewis, Chris wrote:
>
> Hi Scott/Wes,
>
>  
>
> How did you work out the backtrace?  I have a similar issue and was 
> wondering how you knew it was that specific Bug?
>
>  
>
> Chris
>
>  
>
> *From:* cisco-voip-bounces at puck.nether.net 
> <mailto:cisco-voip-bounces at puck.nether.net> 
> [mailto:cisco-voip-bounces at puck.nether.net] *On Behalf Of *Wes Sisk
> *Sent:* 27 January 2010 19:40
> *To:* Voll, Scott
> *Cc:* Cisco VoIP (cisco-voip at puck.nether.net 
> <mailto:cisco-voip at puck.nether.net>)
> *Subject:* Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> RTMT check boxes:
> trace and log central
> collect files
> "select ccm services/applications" -> cisco extended functions, cisco 
> extended functions report
> "select system services/applications" -> cisco corefile recovery tool
>
> /Wes
>
> On Wednesday, January 27, 2010 2:17:24 PM, Voll, Scott 
> <Scott.Voll at wesd.org> <mailto:Scott.Voll at wesd.org> wrote:
>
> Wes---
>
>  
>
> I feel like a dip shmoe for asking.... But how do I get the:
>
>  
>
> Core files and CEF traces?
>
>  
>
> Scott
>
>  
>
> *From:* Wes Sisk [mailto:wsisk at cisco.com]
> *Sent:* Wednesday, January 27, 2010 11:00 AM
> *To:* Voll, Scott
> *Cc:* Cisco VoIP (cisco-voip at puck.nether.net 
> <mailto:cisco-voip at puck.nether.net>)
> *Subject:* Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> Good news: you're not the only one seeing this on 6.1.4 which is 
> supposed to already contain the fix
> bad news: no root cause yet so more investigation is required
>
> best bet is to open a TAC case with the backtrace, version, core file, 
> and cef traces.  it is possible the fix did not get merged properly. 
> but that is still a discussion that needs to come out of a specific 
> tac case.
>
> /Wes
>
> On Wednesday, January 27, 2010 1:42:15 PM, Voll, Scott 
> <Scott.Voll at wesd.org> <mailto:Scott.Voll at wesd.org> wrote:
>
>
> 6.1.4.2000.2
>
>  
>
> *From:* Wes Sisk [mailto:wsisk at cisco.com]
> *Sent:* Wednesday, January 27, 2010 10:40 AM
> *To:* Voll, Scott
> *Cc:* Cisco VoIP (cisco-voip at puck.nether.net 
> <mailto:cisco-voip at puck.nether.net>)
> *Subject:* Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> you didn't include what version you are running ;)
>
> Looks like
> CSCso02381    CEF core due to invalid incoming QBE message header
>
> I just updated the RNE to include the include the backtrace, check bug 
> toolkit in 24-48 hours.
>
> Looks like it's fixed in 6.1(3.3206.1), 6.1(4), and 7.0(1).  What 
> version are you running?
>
> /Wes
>
> On Wednesday, January 27, 2010 12:14:32 PM, Voll, Scott 
> <Scott.Voll at wesd.org> <mailto:Scott.Voll at wesd.org> wrote:
>
>
>
> Where do I find this "backtrace" in the bug toolkit?
>
>  
>
> Here is my backtrace:
>
>  
>
> #0  0x0032ca92 in strncpy () from /lib/tls/libc.so.6
>
> #1  0x00e17f38 in OK_BOD_strncpy () from /lib/csa/libcsa.so.6
>
> #2  0x00e15570 in strncpy () from /lib/csa/libcsa.so.6
>
> #3  0x080b26eb in CQBE_Parser::qbeGet (pMsgBuf_=0x1756f000, 
> pParsedMsg_=0x1755e5ec, pQbeString_=0x2e312e36, 
> pNextOffset_=0x1755e46c) at qbeparser.cpp:410
>
> #4  0x080b4dc6 in CTIQBE::ProviderOpenCompletedEvent::qbeGet 
> (this=0x80b26eb, pMsgBuf_=0x175625c0, pNextOffset_=0x1755e46c) at 
> qbeparser.cpp:2847
>
> #5  0x080b948c in CQBE_Parser::qbeParse (this=0x175625a0, 
> pInputMsg_=0x175625c0, inputMsgLength_=89, pParsedMsg_=0x1755e5a0, 
> parsedMsgBufferSize_=16384) at qbeparser.cpp:5542
>
> #6  0x08095221 in CCtiInterface::receiveLoop (this=0x0) at 
> ctireceiver.cpp:428
>
> #7  0x00000000 in ?? ()
>
>  
>
> Scott
>
>  
>
> *From:* Wes Sisk [mailto:wsisk at cisco.com]
> *Sent:* Wednesday, January 27, 2010 8:46 AM
> *To:* Voll, Scott
> *Cc:* Cisco VoIP (cisco-voip at puck.nether.net 
> <mailto:cisco-voip at puck.nether.net>)
> *Subject:* Re: [cisco-voip] FW: [RTMT-ALERT-CMPUB-Cluster] 
> CoreDumpFileFound
>
>  
>
> yep, it means your cef process generated a core dump.  i.e. it 
> crashed.  this is the equivalent of windows dr.watson and user.dmp.
>
> First level diagnosis:
> from the CLI use 'utils core list' to identify the core file and then 
> 'utils core analyze' to generate the backtrace from the core file.  
> you *should* be able to search for the backtrace in bug toolkit to 
> find the relevant defect. if btk is not cooperating then open a tac 
> case and provide your CM version and the backtrace.  That is adequate 
> to identify the defect for known crashes.
>
> Second level diagnosis:
> If the backtrace does not match a known signature then we have to 
> diagnose the core.  That requires:
> - the actual cef core file (core files may be very large)
> - the backtrace from the core file
> - cef traces for 1 hour leading up to the failure
> - possibly other traces such as cm SDI/SDL for the same timeframe
>
> /Wes
>
>
> On Wednesday, January 27, 2010 10:41:41 AM, Voll, Scott 
> <Scott.Voll at wesd.org> <mailto:Scott.Voll at wesd.org> wrote:
>
>
>
>
> I'm not finding much on a Google search.  Any idea what this means?
>
>  
>
> I'm guessing a Cisco Extended Function crashed on my first node, but 
> not quite sure what needs to be done or if this is a tac case.
>
>  
>
> From a TAC point of view which RTMT logs should I be getting if I need 
> to open a ticket?
>
>  
>
> This is the first time I've seen this error.
>
>  
>
> Thanks
>
>  
>
> Scott
>
>  
>
> PS. CM 6.1.4-2000.2
>
>  
>
> *From:* RTMT_Admin at wesd.org <mailto:RTMT_Admin at wesd.org> 
> [mailto:RTMT_Admin at wesd.org]
> *Sent:* Wednesday, January 27, 2010 6:06 AM
> *To:* Voll, Scott
> *Subject:* [RTMT-ALERT-CMPUB-Cluster] CoreDumpFileFound
>
>  
>
> CoreDumpFileFound TotalCoresFound : 1 CoreDetails : The following 
> lists up to 6 cores dumped by corresponding applications. Core1 : 
> Cisco Extended Functions (core.13530.11.cef.1264601139) AppID : Cisco 
> Log Partition Monitoring Tool ClusterID : NodeID : CM1 . The alarm is 
> generated on Wed Jan 27 06:05:44 PST 2010.
>
>  
>  
>  
>  
>  
>  
>  
>  
>
>
> ------------------------------------------------------------------------
>
>
>  
>  
>  
>  
>  
>  
>   
>  
>  
>   
>  
>  
>   
>  
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net <mailto:cisco-voip at puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-voip
>   
>
>  
>
>  
>
>  
>
>  
>
> The information contained in this message and its attachments 
> is intended only for the private and confidential use of the 
> intended recipient(s).  If you are not the intended recipient 
> (or have received this e-mail in error) please notify the 
> sender immediately and destroy this e-mail. Any unauthorized 
> copying, disclosure or distribution of the material in this e-
> mail is strictly prohibited.
>   
> The information contained in this message and its attachments 
> is intended only for the private and confidential use of the 
> intended recipient(s).  If you are not the intended recipient 
> (or have received this e-mail in error) please notify the 
> sender immediately and destroy this e-mail. Any unauthorized 
> copying, disclosure or distribution of the material in this e-
> mail is strictly prohibited.
>   
>  
>
>
> ------------------------------------------------------------------------
>
>
>  
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net <mailto:cisco-voip at puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-voip
>   
> The information contained in this message and its attachments 
> is intended only for the private and confidential use of the 
> intended recipient(s).  If you are not the intended recipient 
> (or have received this e-mail in error) please notify the 
> sender immediately and destroy this e-mail. Any unauthorized 
> copying, disclosure or distribution of the material in this e-
> mail is strictly prohibited.
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20101118/9accb6ca/attachment.html>


More information about the cisco-voip mailing list