[cisco-voip] CallProcessingNodeCpuPegging

Wed Jul 12 10:13:58 EDT 2006

Any chance you made copies of those files before deleting?  Need to  
check them and see what field has an invalid value and see what the  
permanent fix is.  Temporarily you can modify the bad field and drop  
it back into the cdr directory to allow it to insert.

/Wes
On Jul 11, 2006, at 10:58 AM, Voll, Scott wrote:

I deleted the files in the BAD directory and rebooted and everything  
seems to be better.

The answer to all M$ questions…….. shut up and reboot. ;-)

Scott

From: Erick Bergquist [mailto:erickbe at yahoo.com]
Sent: Tuesday, July 11, 2006 5:14 AM
To: Erick Bergquist; Voll, Scott; Kevin Thorngren
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

One more thing on tracking the bad entries down, is to open a few of  
the flat files in the bad folder and see if you can notice something  
that stands out and maybe pin it down that way. If you want to zip a  
few up and send them to me I can take a look.

Are you using FACs at all? There was a bug where if the FAC  
description had a ' in it this could happen.  The fix for this was to  
edit the FAC descriptions so it had no ' in it.

----- Original Message ----
From: Erick Bergquist <erickbe at yahoo.com>
To: "Voll, Scott" <Scott.Voll at wesd.org>; Kevin Thorngren  
<kthorngr at cisco.com>
Cc: cisco-voip at puck.nether.net
Sent: Monday, July 10, 2006 10:12:22 PM
Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

The CDR files backing up event log error is caused by there being  
more then 200 files in one of those 3 folders (BAD in this case). I  
had this a few months ago and Wes had dug into it and I believe filed  
a ddts to document the cause of the error since it was not documented  
anywhere.

If the BAD records are occurring a few times a day, it may be due to  
invalid data in CDR files (other bugs on these sort of issues) and  
you may want to investigate the bad flat files some more. Look at the  
timestamp on one of them and see if there is a error in event viewer  
application log from that time about CDRServerDown error. Those  
errors can be seen when the flat file can not be inserted for one  
reason or another.

Start by cutting and pasting a few of the bad files from the bad  
folder to the CDR folder and see if they process ok. If you try this  
a few times when it is not busy cpu wise, etc and the same files  
always end up back in the bad folder chances are the flat file has  
data in it the insert process doesn't like.

What I have done to track down the cause is I take the bad flat file  
and I open it in notepad, and open another notepad window and cut and  
paste a few entries from the bad flat file into the other notepad  
window then I save the bad flat file with less entries in it, then I  
move the bad file i edited back to the cdr folder and see if the  
insert process inserts it fine or if it gets put back in bad folder.  
If it goes back in bad folder, i open it again and cut/paste a few  
more lines into the notepad window and save it and move it back until  
it gets inserted. After a couple times of doing this you can get an  
idea of what the bad entry is causing the insert not to go. And see  
what is different about that entry.

The bugs I have seen have been garbage in the cdr flat files, and  
cdrs not being inserted due to commands or aprostophes <sp>, comment  
field being greater then 255 characters, etc.

----- Original Message ----
From: "Voll, Scott" <Scott.Voll at wesd.org>
To: Kevin Thorngren <kthorngr at cisco.com>
Cc: cisco-voip at puck.nether.net
Sent: Monday, July 10, 2006 12:07:04 PM
Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

I see all my calls in CDR, so I believe everything is working correctly.

We are using a cron job to move CDR files to a offbox SQL server to  
do billing.  Could that cause it?  It was done in house.

The BAD folder has multiple files per day from july last year though  
last month.

Thanks

Scott

PS… I’m still rebooting tonight.

From: Kevin Thorngren [mailto:kthorngr at cisco.com]
Sent: Monday, July 10, 2006 9:56 AM
To: Voll, Scott
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

Well, 3000 files shouldn't cause a CPU spike. Obviously the 2695  
files in the BAD folder are causing the event log message you are  
seeing. I think the threshold is 200. It has been a long time since I  
have had to troubleshoot CDRs moving to the BAD folder. CDR Insert  
traces would help but they may need to be set to detailed when the  
next CDR is moved to the BAD folder to find the problem.

Does it seem like these files are all recent and that none of your  
CDRs are being inserted?

I have seen issues were customers using third party CDR tools run  
into issues with the SQL trigger from the third party. This could  
cause the CDRs to move to the BAD folder.

When I was in TAC I never got away with asking the customer to  
reboot ;-)

Based on the info you provided I am not convinced that the CPU spike  
is a result of the CDR Insert issue (although I won't rule anything  
out). The only reason I mentioned troubleshooting before rebooting is  
if CDRs were not being inserted and staying in the CDR folder then  
there would be a possibility of fixing the CDR Insert problem without  
a reboot. Then you could see if the CPU spike problem was resolved.

Kevin

On Jul 10, 2006, at 12:15 PM, Voll, Scott wrote:

Kevin—

Thanks for the reply….. here is what I know.

Pub:

  CDR directory – 0 files / folders

  CMR Dirctory – 0 Files / Folders

  Bad directory – 2695 files

Sub:

  CDR directory – 0 files / folders

  CMR Dirctory – 0 Files / Folders

I just restarted the CDR insert service on the PUB.

Why do I want to trouble shoot further before rebooting?  Seems as  
it’s windows it might be a better solution.

Scott

From: Kevin Thorngren [mailto:kthorngr at cisco.com]

Sent: Monday, July 10, 2006 8:59 AM

To: Voll, Scott

Cc: cisco-voip at puck.nether.net

Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

This may be a separate issue.

If you are getting a build up of thousands of files in one of the  
following directories then you would see the System process spike the  
CPU each time the folder is accessed.

C:\Program Files\Cisco\CallDetail\CDR

C:\Program Files\Cisco\CallDetail\CMR

C:\Program Files\Cisco\CallDetail\BAD

But I would suspect that you would see these spikes throughout the  
day, each time a file is copied over from one of the Subscribers. I  
would recommend waiting until after hours before checking to see how  
many files are in these folders as you run the possibility of spiking  
the CPU if there are many thousands of files.

Maybe a restart of the CDR Insert service would resolve the issue.  
Again I would wait for after hours in case of a CPU spike. You might  
want to enable the CDR Insert traces to troubleshoot the problem  
before doing any of the restarts/reboots.

I would recommend starting a perfmon log of Processor usage to find  
out which process is spiking the CPU.

Kevin

On Jul 10, 2006, at 11:35 AM, Voll, Scott wrote:

I see this in the event log:

Event ID 3

Source:  Cisco Database Layer

Error: kErrorCDRFilesBackingUp - CDR flat files are backing up.

   App ID: Cisco Database Layer Monitor

   Cluster ID: CMPUB-Cluster

   Node ID: CMPUB

Explanation: CDR flat files are not being removed.  On the primary  
CDR server, verify that the InsertCDR service is running and properly  
configured. On a server not the primary, verify that the location for  
collecting CDR files is accessible via the network.

Recommended Action: Set trace for InsertCDR service to detailed and  
look for errors in the trace.  Check enterprise CDR parameters for  
accuracy..

I went into the Services and all look to be running.  So I’m not sure  
what’s up.  I will be rebooting it tonight.

Scott

From: Kevin Thorngren [mailto:kthorngr at cisco.com]

Sent: Monday, July 10, 2006 8:01 AM

To: Voll, Scott

Cc: cisco-voip at puck.nether.net

Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

Do you know which process is pegging the CPU?

Kevin

On Jul 10, 2006, at 10:52 AM, Voll, Scott wrote:

I installed OS 4.2.sr8 and since then I have been getting CPU pegging  
out around the midnight hour.  I believe it’s the CDR flat not  
getting truncated.

Has anyone else seen this?

Scott

_______________________________________________

cisco-voip mailing list

cisco-voip at puck.nether.net

https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://puck.nether.net/pipermail/cisco-voip/attachments/20060712/9e71033e/attachment-0001.html 