[cisco-voip] CallProcessingNodeCpuPegging

Tue Jul 11 10:58:13 EDT 2006

I deleted the files in the BAD directory and rebooted and everything
seems to be better.  

The answer to all M$ questions........ shut up and reboot. ;-)

Scott

________________________________

From: Erick Bergquist [mailto:erickbe at yahoo.com] 
Sent: Tuesday, July 11, 2006 5:14 AM
To: Erick Bergquist; Voll, Scott; Kevin Thorngren
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

One more thing on tracking the bad entries down, is to open a few of the
flat files in the bad folder and see if you can notice something that
stands out and maybe pin it down that way. If you want to zip a few up
and send them to me I can take a look. 

Are you using FACs at all? There was a bug where if the FAC description
had a ' in it this could happen.  The fix for this was to edit the FAC
descriptions so it had no ' in it. 

----- Original Message ----
From: Erick Bergquist <erickbe at yahoo.com>
To: "Voll, Scott" <Scott.Voll at wesd.org>; Kevin Thorngren
<kthorngr at cisco.com>
Cc: cisco-voip at puck.nether.net
Sent: Monday, July 10, 2006 10:12:22 PM
Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

The CDR files backing up event log error is caused by there being more
then 200 files in one of those 3 folders (BAD in this case). I had this
a few months ago and Wes had dug into it and I believe filed a ddts to
document the cause of the error since it was not documented anywhere. 

If the BAD records are occurring a few times a day, it may be due to
invalid data in CDR files (other bugs on these sort of issues) and you
may want to investigate the bad flat files some more. Look at the
timestamp on one of them and see if there is a error in event viewer
application log from that time about CDRServerDown error. Those errors
can be seen when the flat file can not be inserted for one reason or
another.

Start by cutting and pasting a few of the bad files from the bad folder
to the CDR folder and see if they process ok. If you try this a few
times when it is not busy cpu wise, etc and the same files always end up
back in the bad folder chances are the flat file has data in it the
insert process doesn't like.

What I have done to track down the cause is I take the bad flat file and
I open it in notepad, and open another notepad window and cut and paste
a few entries from the bad flat file into the other notepad window then
I save the bad flat file with less entries in it, then I move the bad
file i edited back to the cdr folder and see if the insert process
inserts it fine or if it gets put back in bad folder. If it goes back in
bad folder, i open it again and cut/paste a few more lines into the
notepad window and save it and move it back until it gets inserted.
After a couple times of doing this you can get an idea of what the bad
entry is causing the insert not to go. And see what is different about
that entry. 

The bugs I have seen have been garbage in the cdr flat files, and cdrs
not being inserted due to commands or aprostophes <sp>, comment field
being greater then 255 characters, etc. 

----- Original Message ----
From: "Voll, Scott" <Scott.Voll at wesd.org>
To: Kevin Thorngren <kthorngr at cisco.com>
Cc: cisco-voip at puck.nether.net
Sent: Monday, July 10, 2006 12:07:04 PM
Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

I see all my calls in CDR, so I believe everything is working correctly.

We are using a cron job to move CDR files to a offbox SQL server to do
billing.  Could that cause it?  It was done in house.

The BAD folder has multiple files per day from july last year though
last month.  

Thanks

Scott

PS... I'm still rebooting tonight.

________________________________

From: Kevin Thorngren [mailto:kthorngr at cisco.com] 
Sent: Monday, July 10, 2006 9:56 AM
To: Voll, Scott
Cc: cisco-voip at puck.nether.net
Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging

Well, 3000 files shouldn't cause a CPU spike. Obviously the 2695 files
in the BAD folder are causing the event log message you are seeing. I
think the threshold is 200. It has been a long time since I have had to
troubleshoot CDRs moving to the BAD folder. CDR Insert traces would help
but they may need to be set to detailed when the next CDR is moved to
the BAD folder to find the problem. 

Does it seem like these files are all recent and that none of your CDRs
are being inserted? 

I have seen issues were customers using third party CDR tools run into
issues with the SQL trigger from the third party. This could cause the
CDRs to move to the BAD folder. 

When I was in TAC I never got away with asking the customer to reboot
;-) 

Based on the info you provided I am not convinced that the CPU spike is
a result of the CDR Insert issue (although I won't rule anything out).
The only reason I mentioned troubleshooting before rebooting is if CDRs
were not being inserted and staying in the CDR folder then there would
be a possibility of fixing the CDR Insert problem without a reboot. Then
you could see if the CPU spike problem was resolved. 

Kevin 

On Jul 10, 2006, at 12:15 PM, Voll, Scott wrote: 

	Kevin- 

	Thanks for the reply..... here is what I know. 

	Pub: 

	 CDR directory - 0 files / folders 

	 CMR Dirctory - 0 Files / Folders 

	 Bad directory - 2695 files 

	Sub: 

	 CDR directory - 0 files / folders 

	 CMR Dirctory - 0 Files / Folders 

	I just restarted the CDR insert service on the PUB. 

	Why do I want to trouble shoot further before rebooting?  Seems
as it's windows it might be a better solution. 

	Scott 

	From: Kevin Thorngren [mailto:kthorngr at cisco.com] 

	Sent: Monday, July 10, 2006 8:59 AM 

	To: Voll, Scott 

	Cc: cisco-voip at puck.nether.net 

	Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging 

	This may be a separate issue. 

	If you are getting a build up of thousands of files in one of
the following directories then you would see the System process spike
the CPU each time the folder is accessed. 

	C:\Program Files\Cisco\CallDetail\CDR 

	C:\Program Files\Cisco\CallDetail\CMR 

	C:\Program Files\Cisco\CallDetail\BAD 

	But I would suspect that you would see these spikes throughout
the day, each time a file is copied over from one of the Subscribers. I
would recommend waiting until after hours before checking to see how
many files are in these folders as you run the possibility of spiking
the CPU if there are many thousands of files. 

	Maybe a restart of the CDR Insert service would resolve the
issue. Again I would wait for after hours in case of a CPU spike. You
might want to enable the CDR Insert traces to troubleshoot the problem
before doing any of the restarts/reboots. 

	I would recommend starting a perfmon log of Processor usage to
find out which process is spiking the CPU. 

	Kevin 

	On Jul 10, 2006, at 11:35 AM, Voll, Scott wrote: 

		I see this in the event log: 

		Event ID 3 

		Source:  Cisco Database Layer 

		Error: kErrorCDRFilesBackingUp - CDR flat files are
backing up. 

		  App ID: Cisco Database Layer Monitor 

		  Cluster ID: CMPUB-Cluster 

		  Node ID: CMPUB 

		Explanation: CDR flat files are not being removed.  On
the primary CDR server, verify that the InsertCDR service is running and
properly configured. On a server not the primary, verify that the
location for collecting CDR files is accessible via the network. 

		Recommended Action: Set trace for InsertCDR service to
detailed and look for errors in the trace.  Check enterprise CDR
parameters for accuracy.. 

		I went into the Services and all look to be running.  So
I'm not sure what's up.  I will be rebooting it tonight. 

		Scott 

		From: Kevin Thorngren [mailto:kthorngr at cisco.com] 

		Sent: Monday, July 10, 2006 8:01 AM 

		To: Voll, Scott 

		Cc: cisco-voip at puck.nether.net 

		Subject: Re: [cisco-voip] CallProcessingNodeCpuPegging 

		Do you know which process is pegging the CPU? 

		Kevin 

		On Jul 10, 2006, at 10:52 AM, Voll, Scott wrote: 

			I installed OS 4.2.sr8 and since then I have
been getting CPU pegging out around the midnight hour.  I believe it's
the CDR flat not getting truncated. 

			Has anyone else seen this? 

			Scott 

			_______________________________________________ 

			cisco-voip mailing list 

			cisco-voip at puck.nether.net 

https://puck.nether.net/mailman/listinfo/cisco-voip 

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://puck.nether.net/pipermail/cisco-voip/attachments/20060711/814a11fc/attachment-0001.html 