[cisco-voip] Differences between DRS and CDR SFTP activities

Ryan Ratliff rratliff at cisco.com
Fri Feb 22 16:46:26 EST 2013 

What I would hope to see is the exact sftp operation that was being done, but only in hopes that it would give a clue as to what permissions issue we are getting hit by.

-Ryan

On Feb 22, 2013, at 4:28 PM, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:

not sure what, if any, logs we're going to get from the NetApp. I checked the configuration of the traces and out of Fatal, Error, Warn, Info, Debug; I'm on Info.

I'm sure Debug will give us more data, but will it be useful?

---
Lelio Fulgenzi, B.A.
Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1
(519) 824-4120 x56354 (519) 767-1060 FAX (ANNU)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Cooking with unix is easy. You just sed it and forget it. 
                              - LFJ (with apologies to Mr. Popeil)


From: "Ryan Ratliff" <rratliff at cisco.com>
To: "Lelio Fulgenzi" <lelio at uoguelph.ca>
Cc: "Erick B." <erickbee at gmail.com>, "voip puck" <cisco-voip at puck.nether.net>
Sent: Friday, February 22, 2013 3:14:04 PM
Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities

2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.SFTPConnection (SFTPConnection.java:131) - Sftp Status Exception : com.maverick.sftp.SftpStatusException: Permission denied
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:473) - SFTP error in transferring file to another machine.
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:474) - com.maverick.sftp.SftpStatusException: Permission denied
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:493) - Exception while sending files: com.maverick.sftp.SftpStatusException: Permission denied
2013-02-21 14:02:39,903 ERROR [Thread-10] cdrrep.FtpManager (FtpManager.java:223) - SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 to blackjack.cs.uoguelph.ca

Hard to see exactly what it's doing since I guess these aren't on detailed level.  Any logs from the NetApp to see what it thinks is going on?

-Ryan

On Feb 22, 2013, at 2:21 PM, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:

Well, coincidentally enough, we're using a NetApp. :(

The log entries from the CDR Repository Manager follow. The CDR Agent just has wakeup statements.

We currently have two of billing servers configured, but during troubleshooting, we had to delete/recreate, and I think during this time, it was still trying to clean up the deleted one which is why you see the three destinations.

sending to hiroller succeeds and to blackjack it fails. hiroller is a linux box as well, but it is CIFS mounting a windows server filesystem.

=====
file:cdrrepmgr00094.log

2013-02-21 14:02:29,578 INFO  [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3
2013-02-21 14:02:30,271 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 1
2013-02-21 14:02:33,194 INFO  [Thread-8] cdrrep.FileMonitor (FileMonitor.java:132) - 2 file(s) found under /var/log/active/cm/cdr_repository/tmp
2013-02-21 14:02:33,196 INFO  [Thread-8] cdrrep.FileMonitor (FileMonitor.java:274) - Processed file: /var/log/active/cm/cdr_repository/tmp/cmr_iptccm105-Cluster_07_201302211902_60910
2013-02-21 14:02:33,197 INFO  [Thread-8] cdrrep.FileMonitor (FileMonitor.java:274) - Processed file: /var/log/active/cm/cdr_repository/tmp/cdr_iptccm105-Cluster_07_201302211902_60910
2013-02-21 14:02:35,582 INFO  [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3
2013-02-21 14:02:36,274 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 1
2013-02-21 14:02:36,275 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:345) - CDRREP before sending the file: JVM status: [Max = 227.56MB, Total = 46.44MB, Free = 13.68MB (29.46%), Used = 32.76MB (70.54%)]
2013-02-21 14:02:36,705 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 2
2013-02-21 14:02:36,705 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:166) - There was file delivery failure for destination 2 in last round, re-connect server now!
2013-02-21 14:02:37,302 INFO  [Thread-9] cdrrep.FtpManager (FtpManager.java:227) - FTP success: cmr_iptccm105-Cluster_07_201302211902_60910 to hiroller.cs.uoguelph.ca
2013-02-21 14:02:37,302 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:352) - CDRREP after sending the file: JVM status: [Max = 227.56MB, Total = 46.44MB, Free = 13.47MB (29.01%), Used = 32.97MB (70.99%)]
2013-02-21 14:02:38,788 INFO  [Thread-10] cdrrep.FtpManager (FtpManager.java:78) - FtpManager constructor 1: Establish server connection to host [blackjack.cs.uoguelph.ca] successfully!
2013-02-21 14:02:38,789 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:345) - CDRREP before sending the file: JVM status: [Max = 227.56MB, Total = 46.00MB, Free = 15.44MB (33.56%), Used = 30.56MB (66.44%)]
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.SFTPConnection (SFTPConnection.java:131) - Sftp Status Exception : com.maverick.sftp.SftpStatusException: Permission denied
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:473) - SFTP error in transferring file to another machine.
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:474) - com.maverick.sftp.SftpStatusException: Permission denied
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:493) - Exception while sending files: com.maverick.sftp.SftpStatusException: Permission denied
2013-02-21 14:02:39,903 ERROR [Thread-10] cdrrep.FtpManager (FtpManager.java:223) - SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 to blackjack.cs.uoguelph.ca
2013-02-21 14:02:39,904 ERROR [Thread-10] cdrrep.LogConfig (LogConfig.java:157) - Java Exception happened in my CDRSender, with Exception: 
java.lang.Exception: SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 toblackjack.cs.uoguelph.ca
    at com.cisco.ccm.cdrdlv.cdrrep.FtpManager.sendFile(FtpManager.java:225)
    at com.cisco.ccm.cdrdlv.cdrrep.CDRSender.run(CDRSender.java:349)

2013-02-21 14:02:39,904 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:364) - sendSuccess=false, numOfFailure=5
2013-02-21 14:02:39,904 ERROR [Thread-10] cdrrep.CDRSender (CDRSender.java:365) - java.lang.Exception: SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 toblackjack.cs.uoguelph.ca
2013-02-21 14:02:39,904 ERROR [Thread-10] cdrrep.CDRSender (CDRSender.java:366) - Unable to successfully send file to outside billing server blackjack.cs.uoguelph.ca
2013-02-21 14:02:39,904 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:389) - More than two times of failure, raise alarm
2013-02-21 14:02:39,904 INFO  [Thread-10] alarmutil.AlarmSender (AlarmSender.java:101) - alarmName: CDRFileDeliveryFailureContinues
2013-02-21 14:02:39,904 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:403) -  resendonfailure flag valuetrue
2013-02-21 14:02:39,904 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:406) -  resendonfailure flag valuetrue
2013-02-21 14:02:39,905 INFO  [Thread-10] cdrrep.CDRSender (CDRSender.java:407) -  host valueblackjack.cs.uoguelph.ca
2013-02-21 14:02:41,304 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:345) - CDRREP before sending the file: JVM status: [Max = 227.56MB, Total = 46.00MB, Free = 14.84MB (32.27%), Used = 31.16MB (67.73%)]
2013-02-21 14:02:41,585 INFO  [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3
2013-02-21 14:02:42,367 INFO  [Thread-9] cdrrep.FtpManager (FtpManager.java:227) - FTP success: cdr_iptccm105-Cluster_07_201302211902_60910 to hiroller.cs.uoguelph.ca
2013-02-21 14:02:42,367 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:352) - CDRREP after sending the file: JVM status: [Max = 227.56MB, Total = 46.00MB, Free = 14.61MB (31.77%), Used = 31.39MB (68.23%)]
2013-02-21 14:02:47,588 INFO  [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3
2013-02-21 14:02:48,370 INFO  [Thread-9] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 1

=====

---
Lelio Fulgenzi, B.A.
Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1
(519) 824-4120 x56354 (519) 767-1060 FAX (ANNU)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Cooking with unix is easy. You just sed it and forget it. 
                              - LFJ (with apologies to Mr. Popeil)


From: "Erick B." <erickbee at gmail.com>
To: "Lelio Fulgenzi" <lelio at uoguelph.ca>
Cc: "Ryan Ratliff" <rratliff at cisco.com>, "voip puck" <cisco-voip at puck.nether.net>
Sent: Friday, February 22, 2013 2:14:56 PM
Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities

I have noticed a difference to with the DRS and CDR SFTP jobs, going to a NetApp. DRS worked fine but CDR billing job had issues with NetApp and moved CDR to use Windows server with different SFTP server. Something on my to-do list of things to figure out one of these days...  


On Fri, Feb 22, 2013 at 1:06 PM, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:
hmmm, never thought about collecting those. i'll go look right now. but here's what I see from the syslog host we're sending events to:

Feb 21 13:24:09 iptccm105.cs.uoguelph.ca 2272: : : 697: Feb 21 18:24:09.952 UTC :  %CCM_CDR_REP-CDRREP-3-CDRFileDeliveryFailureContinues: (s)FTP delivery of CDR files failed on retries. BillingServerAddress:blackjack.cs.uoguelph.ca App ID:Cisco CDR Repository Manager Cluster ID: Node ID:iptccm105



---
Lelio Fulgenzi, B.A.
Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1
(519) 824-4120 x56354 (519) 767-1060 FAX (ANNU)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Cooking with unix is easy. You just sed it and forget it. 
                              - LFJ (with apologies to Mr. Popeil)


From: "Ryan Ratliff" <rratliff at cisco.com>
To: "Lelio Fulgenzi" <lelio at uoguelph.ca>
Cc: cisco-voip at puck.nether.net
Sent: Friday, February 22, 2013 2:01:13 PM
Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities


What do the logs for CDR Repository Manager (or CDR Agent, I don't recall offhand which one does the sftp) say about why it's failing?   The only immediate difference I can think of is the nature of the files being written though that doesn't really explain why filesystem ownership has an impact.

-Ryan

On Feb 22, 2013, at 1:42 PM, Lelio Fulgenzi <lelio at uoguelph.ca> wrote:


So, we've set up an SFTP server on a linux box and setup a (mounted) filesystem where a user is not the owner, but part of the group that has rw access. WinSCP works fine, user is able to write files. DRS is fine, backups work fine. CDR billing server does not work, transfers fail with partially written files and then others don't go through at all.

The only way we got CDR transfers to work, was when use used a local filesystem where the user was the owner used a user owned the mounted filesystem it worked. Again, DRS works fine without any of these issues.

Can anyone speak to the nuances between the DRS SFTP process and the CDR SFTP process and why one would work and one would not?



---
Lelio Fulgenzi, B.A.
Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1
(519) 824-4120 x56354 (519) 767-1060 FAX (ANNU)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Cooking with unix is easy. You just sed it and forget it. 
                              - LFJ (with apologies to Mr. Popeil)


_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip


_______________________________________________
cisco-voip mailing list
cisco-voip at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20130222/b4722c3c/attachment.html>

More information about the cisco-voip mailing list