[cisco-voip] CallProcessingNodeCpuPegging

Tue Jul 11 22:16:10 EDT 2006

Keep a eye on the bad folder then... the files may come back slowly if it is a issue with certain call records data cdr insert doesn't like. 

----- Original Message ----
From: "Voll, Scott" <Scott.Voll at wesd.org>
To: Erick Bergquist <erickbe at yahoo.com>; Kevin Thorngren <kthorngr at cisco.com>
Cc: cisco-voip at puck.nether.net
Sent: Tuesday, July 11, 2006 9:58:13 AM
Subject: RE: [cisco-voip] CallProcessingNodeCpuPegging

                 I deleted the files in the BAD directory and rebooted and everything seems to be better.  

   The answer to all M$ questions…….. shut up and reboot. ;-)

   Scott

  From:  Erick Bergquist [mailto:erickbe at yahoo.com] 
 Sent: Tuesday, July 11, 2006 5:14 AM
 To: Erick  Bergquist ; Voll, Scott; Kevin Thorngren
 Cc: cisco-voip at puck.nether.net
 Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

       One more thing on tracking the bad entries down, is to open a few of the flat files in the bad folder and see if you can notice something that stands out and maybe pin it down that way. If you want to zip a few up and send them to me I can take a look. 

 Are you using FACs at all? There was a bug where if the FAC description had a ' in it this could happen.  The fix for this was to edit the FAC descriptions so it had no ' in it. 
     ----- Original Message ----
 From: Erick Bergquist  <erickbe at yahoo.com>
 To: "Voll, Scott" <Scott.Voll at wesd.org>; Kevin Thorngren <kthorngr at cisco.com>
 Cc: cisco-voip at puck.nether.net
 Sent: Monday, July 10, 2006 10:12:22 PM
 Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging
       The CDR files backing up event log error is caused by there being more then 200 files in one of those 3 folders (BAD in this case). I had this a few months ago and Wes had dug into it and I believe filed a ddts to document the cause of the error since it was not documented anywhere. 

 If the BAD records are occurring a few times a day, it may be due to invalid data in CDR files (other bugs on these sort of issues) and you may want to investigate the bad flat files some more. Look at the timestamp on one of them and see if there is a error in event viewer application log from that time about CDRServerDown error. Those errors can be seen when the flat file can not be inserted for one reason or another.

 Start by cutting and pasting a few of the bad files from the bad folder to the CDR folder and see if they process ok. If you try this a few times when it is not busy cpu wise, etc and the same files always end up back in the bad folder chances are the flat file has data in it the insert process doesn't like.

 What I have done to track down the cause is I take the bad flat file and I open it in notepad, and open another notepad window and cut and paste a few entries from the bad flat file into the other notepad window then I save the bad flat file with less entries in it, then I move the bad file i edited back to the cdr folder and see if the insert process inserts it fine or if it gets put back in bad folder. If it goes back in bad folder, i open it again and cut/paste a few more lines into the notepad window and save it and move it back until it gets inserted. After a couple times of doing this you can get an idea of what the bad entry is causing the insert not to go. And see what is different about that entry. 

 The bugs I have seen have been garbage in the cdr flat files, and cdrs not being inserted due to commands or aprostophes <sp>, comment field being greater then 255 characters, etc. 

     ----- Original Message ----
 From: "Voll, Scott" <Scott.Voll at wesd.org>
 To: Kevin Thorngren <kthorngr at cisco.com>
 Cc: cisco-voip at puck.nether.net
 Sent: Monday, July 10, 2006 12:07:04 PM
 Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging
     I see all my calls in CDR, so I believe everything is working correctly.

   We are using a cron job to move CDR files to a offbox SQL server to do billing.  Could that cause it?  It was done in house.

   The BAD folder has multiple files per day from july last year though last month.  

   Thanks

   Scott

   PS… I’m still rebooting tonight.

  From: Kevin Thorngren [mailto:kthorngr at cisco.com] 
 Sent: Monday, July 10, 2006 9:56 AM
 To: Voll, Scott
 Cc: cisco-voip at puck.nether.net
 Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

     Well, 3000 files shouldn't cause a CPU spike. Obviously the 2695 files in the BAD folder are causing the event log message you are seeing. I think the threshold is 200. It has been a long time since I have had to troubleshoot CDRs moving to the BAD folder. CDR Insert traces would help but they may need to be set to detailed when the next CDR is moved to the BAD folder to find the problem. 

     Does it seem like these files are all recent and that none of your CDRs are being inserted? 

     I have seen issues were customers using third party CDR tools run into issues with the SQL trigger from the third party. This could cause the CDRs to move to the BAD folder. 

     When I was in TAC I never got away with asking the customer to reboot ;-) 

     Based on the info you provided I am not convinced that the CPU spike is a result of the CDR Insert issue (although I won't rule anything out). The only reason I mentioned troubleshooting before rebooting is if CDRs were not being inserted and staying in the CDR folder then there would be a possibility of fixing the CDR Insert problem without a reboot. Then you could see if the CPU spike problem was resolved. 

     Kevin 

    On Jul 10, 2006, at 12:15 PM, Voll, Scott wrote: 

     Kevin— 

    Thanks for the reply….. here is what I know. 

    Pub: 

     CDR directory – 0 files / folders 

     CMR Dirctory – 0 Files / Folders 

     Bad directory – 2695 files 

    Sub: 

     CDR directory – 0 files / folders 

     CMR Dirctory – 0 Files / Folders 

    I just restarted the CDR insert service on the PUB. 

    Why do I want to trouble shoot further before rebooting?  Seems as it’s windows it might be a better solution. 

    Scott 

     From: Kevin Thorngren [mailto:kthorngr at cisco.com] 

    Sent: Monday, July 10, 2006 8:59 AM 

    To: Voll, Scott 

    Cc: cisco-voip at puck.nether.net 

    Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging 

    This may be a separate issue. 

    If you are getting a build up of thousands of files in one of the following directories then you would see the System process spike the CPU each time the folder is accessed. 

    C:\Program Files\Cisco\CallDetail\CDR 

    C:\Program Files\Cisco\CallDetail\CMR 

    C:\Program Files\Cisco\CallDetail\BAD 

    But I would suspect that you would see these spikes throughout the day, each time a file is copied over from one of the Subscribers. I would recommend waiting until after hours before checking to see how many files are in these folders as you run the possibility of spiking the CPU if there are many thousands of files. 

    Maybe a restart of the CDR Insert service would resolve the issue. Again I would wait for after hours in case of a CPU spike. You might want to enable the CDR Insert traces to troubleshoot the problem before doing any of the restarts/reboots. 

    I would recommend starting a perfmon log of Processor usage to find out which process is spiking the CPU. 

    Kevin 

    On Jul 10, 2006, at 11:35 AM, Voll, Scott wrote: 

    I see this in the event log: 

    Event ID 3 

    Source:  Cisco Database Layer 

    Error: kErrorCDRFilesBackingUp - CDR flat files are backing up. 

      App ID: Cisco Database Layer Monitor 

      Cluster ID: CMPUB-Cluster 

      Node ID: CMPUB 

    Explanation: CDR flat files are not being removed.  On the primary CDR server, verify that the InsertCDR service is running and properly configured. On a server not the primary, verify that the location for collecting CDR files is accessible via the network. 

    Recommended Action: Set trace for InsertCDR service to detailed and look for errors in the trace.  Check enterprise CDR parameters for accuracy.. 

    I went into the Services and all look to be running.  So I’m not sure what’s up.  I will be rebooting it tonight. 

    Scott 

    From: Kevin Thorngren [mailto:kthorngr at cisco.com] 

    Sent: Monday, July 10, 2006 8:01 AM 

    To: Voll, Scott 

    Cc: cisco-voip at puck.nether.net 

    Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging 

    Do you know which process is pegging the CPU? 

    Kevin 

    On Jul 10, 2006, at 10:52 AM, Voll, Scott wrote: 

    I installed OS 4.2.sr8 and since then I have been getting CPU pegging out around the midnight hour.  I believe it’s the CDR flat not getting truncated. 

    Has anyone else seen this? 

    Scott 

    _______________________________________________ 

    cisco-voip mailing list 

    cisco-voip at puck.nether.net 

    https://puck.nether.net/mailman/listinfo/cisco-voip 

    _______________________________________________
 cisco-voip mailing list
 cisco-voip at puck.nether.net
 https://puck.nether.net/mailman/listinfo/cisco-voip

    _______________________________________________
 cisco-voip mailing list
 cisco-voip at puck.nether.net
 https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://puck.nether.net/pipermail/cisco-voip/attachments/20060711/2226364f/attachment-0001.html 