[cisco-voip] suspected hardware failure

[Ryan Ratliff]

Have you looked at the HP Integrated Management Log?  You may need to download and install the app to read it but if you see any errors in there related to disks, etc you need to get those addressed.

Depending on your CUCM version nonpaged pool memory leaks can also present with very odd performance-like issues and would be cleared by a reboot.

-Ryan

On Apr 14, 2010, at 2:52 PM, Miller, Brinis wrote:

> At the time this was happening it had all the symptoms.
> The CallManager with the IPCC coresident, which is also the TFTP server stalled, while the server with all the errors went slowly down.
> I rebooted the TFTP CallManager with the IPCC and all its functionality came back.
> Meanwhile the error server continued its downward spiral.
> I do not want to relive that day again.
> I have placed a TAC ticket maybe they can shed light on this.
> 
> B 
> 
> 
> -----Original Message-----
> From: Peter Slow [mailto:peter.slow at gmail.com] 
> Sent: Wednesday, April 14, 2010 2:41 PM
> To: Miller, Brinis
> Cc: Dennis Heim; cisco-voip at puck.nether.net
> Subject: Re: [cisco-voip] suspected hardware failure
> 
> it is unlikely that a failing disk drive would be remedied or leave
> its failing state  from a reboot.
> the HP Diagnostic utilities will let you run a report that shows you
> error counters for the drives, i'd recommend looking at that if you're
> concerned. if you were having perfrmance related issues, causing TCD
> heartbeat issues (heartbeat < 24 bpm) you would likely also be seeing
> things liek code yellows, phone unregistrations delayed dial-tone etc.
> unfortunately, these symptoms are all possible with memory leaks and
> high CPU utilization as well. if you _dont_ see any of those issues,
> there's a higher chance it's something related specifically to TCD.
> 
> What i'd recommend is looking at the perfmon counter for TCD to see if
> the heartbeat is "less than 24" bpm, or if it's 0 BPM. one indicates a
> performance issue, the latter indicates your service is just hung.
> 
> -Peter
> 
> On Wed, Apr 14, 2010 at 9:14 AM, Miller, Brinis <bmiller at chesco.org> wrote:
>> Yes I have several good backups.
>> 
>> I do have support.
>> 
>> 
>> 
>> From: Dennis Heim [mailto:Dennis.Heim at cdw.com]
>> Sent: Wednesday, April 14, 2010 9:12 AM
>> To: Miller, Brinis; cisco-voip at puck.nether.net
>> Subject: RE: suspected hardware failure
>> 
>> 
>> 
>> Do you have a good BARS backup?
>> 
>> Is your server covered by a service contract?
>> 
>> 
>> 
>> Dennis Heim
>> Network Voice Engineer
>> CDW  Advanced Technology Services
>> 11711 N. Meridian Street, Suite 225
>> Carmel, IN  46032
>> 
>> 317.569.4255 Office
>> 317.569.4201 Fax
>> 317.694.6070 Cell
>> 
>> dennis.heim at cdw.com
>> www.berbee.com
>> 
>> 
>> 
>> From: Miller, Brinis [mailto:bmiller at chesco.org]
>> Sent: Wednesday, April 14, 2010 9:09 AM
>> To: Dennis Heim; cisco-voip at puck.nether.net
>> Subject: RE: suspected hardware failure
>> 
>> 
>> 
>> Sorry for the lack of info, some people say I give too much. I suppose my
>> question was a little Spartan.
>> 
>> CCM 4.2(3)sr4b
>> 
>> HP Proliant DL380 G4 server. W2000-4-6sr9 platform
>> 
>> Yes, I know that both CTI and TCD are related to AC, but can be affected by
>> the CallManager service not receiving enough cpu time.
>> 
>> This can be caused by a slowing disk or failing array controller, I have
>> read.
>> 
>> The cpu and memory usage, now are normal as before the incident requiring
>> the reboot. Spiking before the reboot.
>> 
>> The CallManager service, set to automatic, had stopped and started a number
>> of times.
>> 
>> The server appears to be running normally now, but had caused a catastrophe.
>> 
>> Please see attached file
>> 
>> Please see link below.  Read to very end.
>> 
>> http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1269279479813+28353475&threadId=180196
>> 
>> 
>> 
>> Thank you for your reply
>> 
>> 
>> 
>> From: Dennis Heim [mailto:Dennis.Heim at cdw.com]
>> Sent: Tuesday, April 13, 2010 5:19 PM
>> To: Miller, Brinis; cisco-voip at puck.nether.net
>> Subject: RE: suspected hardware failure
>> 
>> 
>> 
>> TCD is related to Attendant Console. What is your cpu and memory utilization
>> like? What version of ccm?
>> 
>> 
>> 
>> Dennis Heim
>> Network Voice Engineer
>> CDW  Advanced Technology Services
>> 11711 N. Meridian Street, Suite 225
>> Carmel, IN  46032
>> 
>> 317.569.4255 Office
>> 317.569.4201 Fax
>> 317.694.6070 Cell
>> 
>> dennis.heim at cdw.com
>> www.berbee.com
>> 
>> 
>> 
>> From: cisco-voip-bounces at puck.nether.net
>> [mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Miller, Brinis
>> Sent: Tuesday, April 13, 2010 3:54 PM
>> To: cisco-voip at puck.nether.net
>> Subject: [cisco-voip] suspected hardware failure
>> 
>> 
>> 
>> Hello,
>> 
>> 
>> 
>> I'm a new comer to the net and would like to post a question.
>> 
>> 
>> 
>> I have a MCS 7835 server which I believe has a hardware failure about to
>> happen.
>> 
>> 
>> 
>> I got a lot of the cpqcissm event 9 messages before it was rebooted. Since
>> then none has reappeared.
>> 
>> I also got
>> 
>> Tcd Server heartbeat rate below 24 beats per minute.
>> 
>> Current heartbeat rate is 0 beats per minute.
>> 
>> I'm thinking I have a hard drive about to bite the dust, how can I test to
>> isolate the cause?
>> 
>> 
>> 
>> Regards,
>> 
>> BMiller
>> 
>> 
>> 
>> 
>> 
>> This County of Chester e-mail message, including any attachments, is
>> intended for the sole use of the individual(s) and entity(ies) to whom it is
>> addressed, and may contain information that is privileged, confidential and
>> exempt from disclosure under applicable law. If you are not the intended
>> addressee, nor authorized to receive for the intended addressee, you are
>> hereby notified that you may not use, copy, disclose or distribute to anyone
>> this e-mail message including any attachments, or any information contained
>> in this e-mail message including any attachments. If you have received this
>> e-mail message in error, please immediately notify the sender by reply
>> e-mail and delete the message. Thank you very much.
>> 
>> 
>> 
>> _______________________________________________
>> cisco-voip mailing list
>> cisco-voip at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-voip
>> 
>> 
> 
> 
> This County of Chester e-mail message, including any attachments, is intended for the sole use of the individual(s) and entity(ies) to whom it is addressed, and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If you are not the intended addressee, nor authorized to receive for the intended addressee, you are hereby notified that you may not use, copy, disclose or distribute to anyone this e-mail message including any attachments, or any information contained in this e-mail message including any attachments. If you have received this e-mail message in error, please immediately notify the sender by reply e-mail and delete the message. Thank you very much.
> 
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip