[cisco-voip] CallProcessingNodeCpuPegging

Tue Jul 11 08:13:34 EDT 2006

One more thing on tracking the bad entries down, is to open a few of the flat files in the bad folder and see if you can notice something that stands out and maybe pin it down that way. If you want to zip a few up and send them to me I can take a look. 

Are you using FACs at all? There was a bug where if the FAC description had a ' in it this could happen.  The fix for this was to edit the FAC descriptions so it had no ' in it. 

----- Original Message ----
From: Erick Bergquist <erickbe at yahoo.com>
To: "Voll, Scott" <Scott.Voll at wesd.org>; Kevin Thorngren <kthorngr at cisco.com>
Cc: cisco-voip at puck.nether.net
Sent: Monday, July 10, 2006 10:12:22 PM
Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

     I see all my calls in CDR, so I believe everything is working correctly.

   We are using a cron job to move CDR files to a offbox SQL server to do billing.  Could that cause it?  It was done in house.

   The BAD folder has multiple files per day from july last year though last month.  

   Thanks

   Scott

   PS… I’m still rebooting tonight.

  From: Kevin Thorngren [mailto:kthorngr at cisco.com] 
 Sent: Monday, July 10, 2006 9:56 AM
 To: Voll, Scott
 Cc: cisco-voip at puck.nether.net
 Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

     Well, 3000 files shouldn't cause a CPU spike. Obviously the 2695 files in the BAD folder are causing the event log message you are seeing. I think the threshold is 200. It has been a long time since I have had to troubleshoot CDRs moving to the BAD folder. CDR Insert traces would help but they may need to be set to detailed when the next CDR is moved to the BAD folder to find the problem. 

     Does it seem like these files are all recent and that none of your CDRs are being inserted? 

     I have seen issues were customers using third party CDR tools run into issues with the SQL trigger from the third party. This could cause the CDRs to move to the BAD folder. 

     When I was in TAC I never got away with asking the customer to reboot ;-) 

     Based on the info you provided I am not convinced that the CPU spike is a result of the CDR Insert issue (although I won't rule anything out). The only reason I mentioned troubleshooting before rebooting is if CDRs were not being inserted and staying in the CDR folder then there would be a possibility of fixing the CDR Insert problem without a reboot. Then you could see if the CPU spike problem was resolved. 

     Kevin 

    On Jul 10, 2006, at 12:15 PM, Voll, Scott wrote: 

     Kevin— 

    Thanks for the reply….. here is what I know. 

    Pub: 

     CDR directory – 0 files / folders 

     CMR Dirctory – 0 Files / Folders 

     Bad directory – 2695 files 

    Sub: 

     CDR directory – 0 files / folders 

     CMR Dirctory – 0 Files / Folders 

    I just restarted the CDR insert service on the PUB. 

    Why do I want to trouble shoot further before rebooting?  Seems as it’s windows it might be a better solution. 

    Scott 

     From: Kevin Thorngren [mailto:kthorngr at cisco.com] 

    Sent: Monday, July 10, 2006 8:59 AM 

    To: Voll, Scott 

    Cc: cisco-voip at puck.nether.net 

    Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging 

    This may be a separate issue. 

    If you are getting a build up of thousands of files in one of the following directories then you would see the System process spike the CPU each time the folder is accessed. 

    C:\Program Files\Cisco\CallDetail\CDR 

    C:\Program Files\Cisco\CallDetail\CMR 

    C:\Program Files\Cisco\CallDetail\BAD 

    But I would suspect that you would see these spikes throughout the day, each time a file is copied over from one of the Subscribers. I would recommend waiting until after hours before checking to see how many files are in these folders as you run the possibility of spiking the CPU if there are many thousands of files. 

    Maybe a restart of the CDR Insert service would resolve the issue. Again I would wait for after hours in case of a CPU spike. You might want to enable the CDR Insert traces to troubleshoot the problem before doing any of the restarts/reboots. 

    I would recommend starting a perfmon log of Processor usage to find out which process is spiking the CPU. 

    Kevin 

    On Jul 10, 2006, at 11:35 AM, Voll, Scott wrote: 

    I see this in the event log: 

    Event ID 3 

    Source:  Cisco Database Layer 

    Error: kErrorCDRFilesBackingUp - CDR flat files are backing up. 

      App ID: Cisco Database Layer Monitor 

      Cluster ID: CMPUB-Cluster 

      Node ID: CMPUB 

    Explanation: CDR flat files are not being removed.  On the primary CDR server, verify that the InsertCDR service is running and properly configured. On a server not the primary, verify that the location for collecting CDR files is accessible via the network. 

    Recommended Action: Set trace for InsertCDR service to detailed and look for errors in the trace.  Check enterprise CDR parameters for accuracy.. 

    I went into the Services and all look to be running.  So I’m not sure what’s up.  I will be rebooting it tonight. 

    Scott 

    From: Kevin Thorngren [mailto:kthorngr at cisco.com] 

    Sent: Monday, July 10, 2006 8:01 AM 

    To: Voll, Scott 

    Cc: cisco-voip at puck.nether.net 

    Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging 

    Do you know which process is pegging the CPU? 

    Kevin 

    On Jul 10, 2006, at 10:52 AM, Voll, Scott wrote: 

    I installed OS 4.2.sr8 and since then I have been getting CPU pegging out around the midnight hour.  I believe it’s the CDR flat not getting truncated. 

    Has anyone else seen this? 

    Scott 

    _______________________________________________ 

    cisco-voip mailing list 

    cisco-voip at puck.nether.net 

    https://puck.nether.net/mailman/listinfo/cisco-voip 

     _______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://puck.nether.net/pipermail/cisco-voip/attachments/20060711/f7db0dac/attachment-0001.html 