[cisco-voip] Differences between DRS and CDR SFTP activities

Sat Feb 23 00:32:09 EST 2013

It's interesting that you raise this point Kiran, because it was due to these errors that we were forced to migrate away from our original SFTP server using GlobalScape EFt server. 

We saw errors with our DRS backups and when testing with a standalone client, we got a similar chmod error. However, during this time, we did not get any CDR transfer errors at all. 

We moved from the GlobalScape SFTP server to a intermediary Linux based SFTP server to try and alleviate these backup errors. CDR transfers continued to work after migrating, but backups continued to cause problems. But different ones, these errors were timeout errors. They happened sporadically. Sometimes once a week, sometimes more, sometimes less. 

Because the server we were using was an old server, rather than troubleshoot, we decided to build a new one. 

And this is where we find ourselves now. 

So based on our experience, I am not seeing the same chmod errors as I was, so I'm thinking it's something else. 

But I could be off my rocker. 

Any more feedback would be great. 

Sent from my iPhone...

"There's no place like 127.0.0.1"

On Feb 23, 2013, at 12:07 AM, "Kiran Ananthpur Bacche (kbacche)" <kbacche at cisco.com> wrote:

> The SFTP Client on the CUCM sends a “CHMOD” operation after transferring the files to the billing server.
> Looks like this operation is failing on the remote billing server resulting in “permission denied” operation.
>  
> So probably the SFTP Server on the billing server is not able to CHMOD the files it has just received and stored on its local system.
>  
>  
> From: cisco-voip-bounces at puck.nether.net [mailto:cisco-voip-bounces at puck.nether.net] On Behalf Of Lelio Fulgenzi
> Sent: Saturday, February 23, 2013 3:18 AM
> To: Ryan Ratliff (rratliff)
> Cc: voip puck
> Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities
>  
> ok. i'll try that sometime next week. can't hurt.
> 
> we've got a few other tests we'd like to try as well. 
> 
> ---
> Lelio Fulgenzi, B.A.
> Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1
> (519) 824-4120 x56354 (519) 767-1060 FAX (ANNU)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Cooking with unix is easy. You just sed it and forget it. 
>                               - LFJ (with apologies to Mr. Popeil)
> 
> 
> From: "Ryan Ratliff" <rratliff at cisco.com>
> To: "Lelio Fulgenzi" <lelio at uoguelph.ca>
> Cc: "Erick B." <erickbee at gmail.com>, "voip puck" <cisco-voip at puck.nether.net>
> Sent: Friday, February 22, 2013 4:46:26 PM
> Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities
> 
> What I would hope to see is the exact sftp operation that was being done, but only in hopes that it would give a clue as to what permissions issue we are getting hit by.
>  
> -Ryan
>  
> On Feb 22, 2013, at 4:28 PM, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:
>  
> not sure what, if any, logs we're going to get from the NetApp. I checked the configuration of the traces and out of Fatal, Error, Warn, Info, Debug; I'm on Info.
> 
> I'm sure Debug will give us more data, but will it be useful?
> 
> ---
> Lelio Fulgenzi, B.A.
> Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1
> (519) 824-4120 x56354 (519) 767-1060 FAX (ANNU)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Cooking with unix is easy. You just sed it and forget it. 
>                               - LFJ (with apologies to Mr. Popeil)
> 
> 
> From: "Ryan Ratliff" <rratliff at cisco.com>
> To: "Lelio Fulgenzi" <lelio at uoguelph.ca>
> Cc: "Erick B." <erickbee at gmail.com>, "voip puck" <cisco-voip at puck.nether.net>
> Sent: Friday, February 22, 2013 3:14:04 PM
> Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities
> 
> 2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.SFTPConnection (SFTPConnection.java:131) - Sftp Status Exception : com.maverick.sftp.SftpStatusException: Permission denied
> 2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:473) - SFTP error in transferring file to another machine.
> 2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:474) - com.maverick.sftp.SftpStatusException: Permission denied
> 2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:493) - Exception while sending files: com.maverick.sftp.SftpStatusException: Permission denied
> 2013-02-21 14:02:39,903 ERROR [Thread-10] cdrrep.FtpManager (FtpManager.java:223) - SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 to blackjack.cs.uoguelph.ca
>  
> Hard to see exactly what it's doing since I guess these aren't on detailed level.  Any logs from the NetApp to see what it thinks is going on?
>  
> -Ryan
>  
> On Feb 22, 2013, at 2:21 PM, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:
>  
> Well, coincidentally enough, we're using a NetApp. :(
> 
> The log entries from the CDR Repository Manager follow. The CDR Agent just has wakeup statements.
> 
> We currently have two of billing servers configured, but during troubleshooting, we had to delete/recreate, and I think during this time, it was still trying to clean up the deleted one which is why you see the three destinations.
> 
> sending to hiroller succeeds and to blackjack it fails. hiroller is a linux box as well, but it is CIFS mounting a windows server filesystem.
> 
> =====
> file:cdrrepmgr00094.log
> 
> 2013-02-21 14:02:29,578 INFO  [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3
> 2013-02-21 14:02:30,271 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 1
> 2013-02-21 14:02:33,194 INFO  [Thread-8] cdrrep.FileMonitor (FileMonitor.java:132) - 2 file(s) found under /var/log/active/cm/cdr_repository/tmp
> 2013-02-21 14:02:33,196 INFO  [Thread-8] cdrrep.FileMonitor (FileMonitor.java:274) - Processed file: /var/log/active/cm/cdr_repository/tmp/cmr_iptccm105-Cluster_07_201302211902_60910
> 2013-02-21 14:02:33,197 INFO  [Thread-8] cdrrep.FileMonitor (FileMonitor.java:274) - Processed file: /var/log/active/cm/cdr_repository/tmp/cdr_iptccm105-Cluster_07_201302211902_60910
> 2013-02-21 14:02:35,582 INFO  [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3
> 2013-02-21 14:02:36,274 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 1
> 2013-02-21 14:02:36,275 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:345) - CDRREP before sending the file: JVM status: [Max = 227.56MB, Total = 46.44MB, Free = 13.68MB (29.46%), Used = 32.76MB (70.54%)]
> 2013-02-21 14:02:36,705 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 2
> 2013-02-21 14:02:36,705 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:166) - There was file delivery failure for destination 2 in last round, re-connect server now!
> 2013-02-21 14:02:37,302 INFO  [Thread-9] cdrrep.FtpManager (FtpManager.java:227) - FTP success: cmr_iptccm105-Cluster_07_201302211902_60910 to hiroller.cs.uoguelph.ca
> 2013-02-21 14:02:37,302 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:352) - CDRREP after sending the file: JVM status: [Max = 227.56MB, Total = 46.44MB, Free = 13.47MB (29.01%), Used = 32.97MB (70.99%)]
> 2013-02-21 14:02:38,788 INFO  [Thread-10] cdrrep.FtpManager (FtpManager.java:78) - FtpManager constructor 1: Establish server connection to host [blackjack.cs.uoguelph.ca] successfully!
> 2013-02-21 14:02:38,789 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:345) - CDRREP before sending the file: JVM status: [Max = 227.56MB, Total = 46.00MB, Free = 15.44MB (33.56%), Used = 30.56MB (66.44%)]
> 2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.SFTPConnection (SFTPConnection.java:131) - Sftp Status Exception : com.maverick.sftp.SftpStatusException: Permission denied
> 2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:473) - SFTP error in transferring file to another machine.
> 2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:474) - com.maverick.sftp.SftpStatusException: Permission denied
> 2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:493) - Exception while sending files: com.maverick.sftp.SftpStatusException: Permission denied
> 2013-02-21 14:02:39,903 ERROR [Thread-10] cdrrep.FtpManager (FtpManager.java:223) - SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 to blackjack.cs.uoguelph.ca
> 2013-02-21 14:02:39,904 ERROR [Thread-10] cdrrep.LogConfig (LogConfig.java:157) - Java Exception happened in my CDRSender, with Exception: 
> java.lang.Exception: SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 toblackjack.cs.uoguelph.ca
>     at com.cisco.ccm.cdrdlv.cdrrep.FtpManager.sendFile(FtpManager.java:225)
>     at com.cisco.ccm.cdrdlv.cdrrep.CDRSender.run(CDRSender.java:349)
> 
> 2013-02-21 14:02:39,904 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:364) - sendSuccess=false, numOfFailure=5
> 2013-02-21 14:02:39,904 ERROR [Thread-10] cdrrep.CDRSender (CDRSender.java:365) - java.lang.Exception: SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 toblackjack.cs.uoguelph.ca
> 2013-02-21 14:02:39,904 ERROR [Thread-10] cdrrep.CDRSender (CDRSender.java:366) - Unable to successfully send file to outside billing server blackjack.cs.uoguelph.ca
> 2013-02-21 14:02:39,904 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:389) - More than two times of failure, raise alarm
> 2013-02-21 14:02:39,904 INFO  [Thread-10] alarmutil.AlarmSender (AlarmSender.java:101) - alarmName: CDRFileDeliveryFailureContinues
> 2013-02-21 14:02:39,904 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:403) -  resendonfailure flag valuetrue
> 2013-02-21 14:02:39,904 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:406) -  resendonfailure flag valuetrue
> 2013-02-21 14:02:39,905 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:407) -  host valueblackjack.cs.uoguelph.ca
> 2013-02-21 14:02:41,304 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:345) - CDRREP before sending the file: JVM status: [Max = 227.56MB, Total = 46.00MB, Free = 14.84MB (32.27%), Used = 31.16MB (67.73%)]
> 2013-02-21 14:02:41,585 INFO  [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3
> 2013-02-21 14:02:42,367 INFO  [Thread-9] cdrrep.FtpManager (FtpManager.java:227) - FTP success: cdr_iptccm105-Cluster_07_201302211902_60910 to hiroller.cs.uoguelph.ca
> 2013-02-21 14:02:42,367 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:352) - CDRREP after sending the file: JVM status: [Max = 227.56MB, Total = 46.00MB, Free = 14.61MB (31.77%), Used = 31.39MB (68.23%)]
> 2013-02-21 14:02:47,588 INFO  [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3
> 2013-02-21 14:02:48,370 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 1
> 
> =====
> 
> ---
> Lelio Fulgenzi, B.A.
> Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1
> (519) 824-4120 x56354 (519) 767-1060 FAX (ANNU)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Cooking with unix is easy. You just sed it and forget it. 
>                               - LFJ (with apologies to Mr. Popeil)
> 
> 
> From: "Erick B." <erickbee at gmail.com>
> To: "Lelio Fulgenzi" <lelio at uoguelph.ca>
> Cc: "Ryan Ratliff" <rratliff at cisco.com>, "voip puck" <cisco-voip at puck.nether.net>
> Sent: Friday, February 22, 2013 2:14:56 PM
> Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities
> 
> I have noticed a difference to with the DRS and CDR SFTP jobs, going to a NetApp. DRS worked fine but CDR billing job had issues with NetApp and moved CDR to use Windows server with different SFTP server. Something on my to-do list of things to figure out one of these days...  
>  
> 
> On Fri, Feb 22, 2013 at 1:06 PM, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:
> hmmm, never thought about collecting those. i'll go look right now. but here's what I see from the syslog host we're sending events to:
> 
> Feb 21 13:24:09 iptccm105.cs.uoguelph.ca 2272: : : 697: Feb 21 18:24:09.952 UTC :  %CCM_CDR_REP-CDRREP-3-CDRFileDeliveryFailureContinues: (s)FTP delivery of CDR files failed on retries. BillingServerAddress:blackjack.cs.uoguelph.ca App ID:Cisco CDR Repository Manager Cluster ID: Node ID:iptccm105
> 
> 
> 
> ---
> Lelio Fulgenzi, B.A.
> Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1
> (519) 824-4120 x56354 (519) 767-1060 FAX (ANNU)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Cooking with unix is easy. You just sed it and forget it. 
>                               - LFJ (with apologies to Mr. Popeil)
> 
> 
> From: "Ryan Ratliff" <rratliff at cisco.com>
> To: "Lelio Fulgenzi" <lelio at uoguelph.ca>
> Cc: cisco-voip at puck.nether.net
> Sent: Friday, February 22, 2013 2:01:13 PM
> Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities
> 
> 
> What do the logs for CDR Repository Manager (or CDR Agent, I don't recall offhand which one does the sftp) say about why it's failing?   The only immediate difference I can think of is the nature of the files being written though that doesn't really explain why filesystem ownership has an impact.
>  
> -Ryan
>  
> On Feb 22, 2013, at 1:42 PM, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:
>  
> 
> So, we've set up an SFTP server on a linux box and setup a (mounted) filesystem where a user is not the owner, but part of the group that has rw access. WinSCP works fine, user is able to write files. DRS is fine, backups work fine. CDR billing server does not work, transfers fail with partially written files and then others don't go through at all.
> 
> The only way we got CDR transfers to work, was when use used a local filesystem where the user was the owner used a user owned the mounted filesystem it worked. Again, DRS works fine without any of these issues.
> 
> Can anyone speak to the nuances between the DRS SFTP process and the CDR SFTP process and why one would work and one would not?
> 
> 
> 
> ---
> Lelio Fulgenzi, B.A.
> Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1
> (519) 824-4120 x56354 (519) 767-1060 FAX (ANNU)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Cooking with unix is easy. You just sed it and forget it. 
>                               - LFJ (with apologies to Mr. Popeil)
> 
> 
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>  
> 
> _______________________________________________
> cisco-voip mailing list
> cisco-voip at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20130223/93d9fe94/attachment.html>