[cisco-voip] Differences between DRS and CDR SFTP activities

Lelio Fulgenzi lelio at uoguelph.ca
Fri Feb 22 16:48:00 EST 2013 

ok. i'll try that sometime next week. can't hurt. 

we've got a few other tests we'd like to try as well. 

--- 
Lelio Fulgenzi, B.A. 
Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1 
(519) 824-4120 x56354 (519) 767-1060 FAX (ANNU) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
Cooking with unix is easy. You just sed it and forget it. 
- LFJ (with apologies to Mr. Popeil) 


----- Original Message -----
From: "Ryan Ratliff" <rratliff at cisco.com> 
To: "Lelio Fulgenzi" <lelio at uoguelph.ca> 
Cc: "Erick B." <erickbee at gmail.com>, "voip puck" <cisco-voip at puck.nether.net> 
Sent: Friday, February 22, 2013 4:46:26 PM 
Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities 

What I would hope to see is the exact sftp operation that was being done, but only in hopes that it would give a clue as to what permissions issue we are getting hit by. 


-Ryan 


On Feb 22, 2013, at 4:28 PM, Lelio Fulgenzi < lelio at uoguelph.ca > wrote: 


not sure what, if any, logs we're going to get from the NetApp. I checked the configuration of the traces and out of Fatal, Error, Warn, Info, Debug; I'm on Info. 

I'm sure Debug will give us more data, but will it be useful? 

--- 
Lelio Fulgenzi, B.A. 
Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1 
(519) 824-4120 x56354 (519) 767-1060 FAX (ANNU) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
Cooking with unix is easy. You just sed it and forget it. 
- LFJ (with apologies to Mr. Popeil) 


----- Original Message -----
From: "Ryan Ratliff" < rratliff at cisco.com > 
To: "Lelio Fulgenzi" < lelio at uoguelph.ca > 
Cc: "Erick B." < erickbee at gmail.com >, "voip puck" < cisco-voip at puck.nether.net > 
Sent: Friday, February 22, 2013 3:14:04 PM 
Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities 



2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.SFTPConnection (SFTPConnection.java:131) - Sftp Status Exception : com.maverick.sftp.SftpStatusException: Permission denied 
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:473) - SFTP error in transferring file to another machine. 
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:474) - com.maverick.sftp.SftpStatusException: Permission denied 
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:493) - Exception while sending files: com.maverick.sftp.SftpStatusException: Permission denied 
2013-02-21 14:02:39,903 ERROR [Thread-10] cdrrep.FtpManager (FtpManager.java:223) - SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 to blackjack.cs.uoguelph.ca 

Hard to see exactly what it's doing since I guess these aren't on detailed level. Any logs from the NetApp to see what it thinks is going on? 


-Ryan 


On Feb 22, 2013, at 2:21 PM, Lelio Fulgenzi < lelio at uoguelph.ca > wrote: 


Well, coincidentally enough, we're using a NetApp. :( 

The log entries from the CDR Repository Manager follow. The CDR Agent just has wakeup statements. 

We currently have two of billing servers configured, but during troubleshooting, we had to delete/recreate, and I think during this time, it was still trying to clean up the deleted one which is why you see the three destinations. 

sending to hiroller succeeds and to blackjack it fails. hiroller is a linux box as well, but it is CIFS mounting a windows server filesystem. 

===== 
file:cdrrepmgr00094.log 

2013-02-21 14:02:29,578 INFO [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3 
2013-02-21 14:02:30,271 INFO [Thread-9] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 1 
2013-02-21 14:02:33,194 INFO [Thread-8] cdrrep.FileMonitor (FileMonitor.java:132) - 2 file(s) found under /var/log/active/cm/cdr_repository/tmp 
2013-02-21 14:02:33,196 INFO [Thread-8] cdrrep.FileMonitor (FileMonitor.java:274) - Processed file: /var/log/active/cm/cdr_repository/tmp/cmr_iptccm105-Cluster_07_201302211902_60910 
2013-02-21 14:02:33,197 INFO [Thread-8] cdrrep.FileMonitor (FileMonitor.java:274) - Processed file: /var/log/active/cm/cdr_repository/tmp/cdr_iptccm105-Cluster_07_201302211902_60910 
2013-02-21 14:02:35,582 INFO [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3 
2013-02-21 14:02:36,274 INFO [Thread-9] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 1 
2013-02-21 14:02:36,275 INFO [Thread-9] cdrrep.CDRSender (CDRSender.java:345) - CDRREP before sending the file: JVM status: [Max = 227.56MB, Total = 46.44MB, Free = 13.68MB (29.46%), Used = 32.76MB (70.54%)] 
2013-02-21 14:02:36,705 INFO [Thread-10] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 2 
2013-02-21 14:02:36,705 INFO [Thread-10] cdrrep.CDRSender (CDRSender.java:166) - There was file delivery failure for destination 2 in last round, re-connect server now! 
2013-02-21 14:02:37,302 INFO [Thread-9] cdrrep.FtpManager (FtpManager.java:227) - FTP success: cmr_iptccm105-Cluster_07_201302211902_60910 to hiroller.cs.uoguelph.ca 
2013-02-21 14:02:37,302 INFO [Thread-9] cdrrep.CDRSender (CDRSender.java:352) - CDRREP after sending the file: JVM status: [Max = 227.56MB, Total = 46.44MB, Free = 13.47MB (29.01%), Used = 32.97MB (70.99%)] 
2013-02-21 14:02:38,788 INFO [Thread-10] cdrrep.FtpManager (FtpManager.java:78) - FtpManager constructor 1: Establish server connection to host [ blackjack.cs.uoguelph.ca ] successfully! 
2013-02-21 14:02:38,789 INFO [Thread-10] cdrrep.CDRSender (CDRSender.java:345) - CDRREP before sending the file: JVM status: [Max = 227.56MB, Total = 46.00MB, Free = 15.44MB (33.56%), Used = 30.56MB (66.44%)] 
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.SFTPConnection (SFTPConnection.java:131) - Sftp Status Exception : com.maverick.sftp.SftpStatusException: Permission denied 
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:473) - SFTP error in transferring file to another machine. 
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:474) - com.maverick.sftp.SftpStatusException: Permission denied 
2013-02-21 14:02:39,903 ERROR [Thread-10] sftpapi.ftpClient (ftpClient.java:493) - Exception while sending files: com.maverick.sftp.SftpStatusException: Permission denied 
2013-02-21 14:02:39,903 ERROR [Thread-10] cdrrep.FtpManager (FtpManager.java:223) - SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 to blackjack.cs.uoguelph.ca 
2013-02-21 14:02:39,904 ERROR [Thread-10] cdrrep.LogConfig (LogConfig.java:157) - Java Exception happened in my CDRSender, with Exception: 
java.lang.Exception: SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 to blackjack.cs.uoguelph.ca 
at com.cisco.ccm.cdrdlv.cdrrep.FtpManager.sendFile(FtpManager.java:225) 
at com.cisco.ccm.cdrdlv.cdrrep.CDRSender.run(CDRSender.java:349) 

2013-02-21 14:02:39,904 INFO [Thread-10] cdrrep.CDRSender (CDRSender.java:364) - sendSuccess=false, numOfFailure=5 
2013-02-21 14:02:39,904 ERROR [Thread-10] cdrrep.CDRSender (CDRSender.java:365) - java.lang.Exception: SFTP/FTP failed: cmr_iptccm105-Cluster_04_201302211900_88292 to blackjack.cs.uoguelph.ca 
2013-02-21 14:02:39,904 ERROR [Thread-10] cdrrep.CDRSender (CDRSender.java:366) - Unable to successfully send file to outside billing server blackjack.cs.uoguelph.ca 
2013-02-21 14:02:39,904 INFO [Thread-10] cdrrep.CDRSender (CDRSender.java:389) - More than two times of failure, raise alarm 
2013-02-21 14:02:39,904 INFO [Thread-10] alarmutil.AlarmSender (AlarmSender.java:101) - alarmName: CDRFileDeliveryFailureContinues 
2013-02-21 14:02:39,904 INFO [Thread-10] cdrrep.CDRSender (CDRSender.java:403) - resendonfailure flag valuetrue 
2013-02-21 14:02:39,904 INFO [Thread-10] cdrrep.CDRSender (CDRSender.java:406) - resendonfailure flag valuetrue 
2013-02-21 14:02:39,905 INFO [Thread-10] cdrrep.CDRSender (CDRSender.java:407) - host value blackjack.cs.uoguelph.ca 
2013-02-21 14:02:41,304 INFO [Thread-9] cdrrep.CDRSender (CDRSender.java:345) - CDRREP before sending the file: JVM status: [Max = 227.56MB, Total = 46.00MB, Free = 14.84MB (32.27%), Used = 31.16MB (67.73%)] 
2013-02-21 14:02:41,585 INFO [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3 
2013-02-21 14:02:42,367 INFO [Thread-9] cdrrep.FtpManager (FtpManager.java:227) - FTP success: cdr_iptccm105-Cluster_07_201302211902_60910 to hiroller.cs.uoguelph.ca 
2013-02-21 14:02:42,367 INFO [Thread-9] cdrrep.CDRSender (CDRSender.java:352) - CDRREP after sending the file: JVM status: [Max = 227.56MB, Total = 46.00MB, Free = 14.61MB (31.77%), Used = 31.39MB (68.23%)] 
2013-02-21 14:02:47,588 INFO [Thread-11] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 3 
2013-02-21 14:02:48,370 INFO [Thread-9] cdrrep.CDRSender (CDRSender.java:105) - CDRSender wakes up for destination 1 

===== 

--- 
Lelio Fulgenzi, B.A. 
Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1 
(519) 824-4120 x56354 (519) 767-1060 FAX (ANNU) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
Cooking with unix is easy. You just sed it and forget it. 
- LFJ (with apologies to Mr. Popeil) 


----- Original Message -----
From: "Erick B." < erickbee at gmail.com > 
To: "Lelio Fulgenzi" < lelio at uoguelph.ca > 
Cc: "Ryan Ratliff" < rratliff at cisco.com >, "voip puck" < cisco-voip at puck.nether.net > 
Sent: Friday, February 22, 2013 2:14:56 PM 
Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities 


I have noticed a difference to with the DRS and CDR SFTP jobs, going to a NetApp. DRS worked fine but CDR billing job had issues with NetApp and moved CDR to use Windows server with different SFTP server. Something on my to-do list of things to figure out one of these days... 



On Fri, Feb 22, 2013 at 1:06 PM, Lelio Fulgenzi < lelio at uoguelph.ca > wrote: 




hmmm, never thought about collecting those. i'll go look right now. but here's what I see from the syslog host we're sending events to: 

Feb 21 13:24:09 iptccm105.cs.uoguelph.ca 2272: : : 697: Feb 21 18:24:09.952 UTC : %CCM_CDR_REP-CDRREP-3-CDRFileDeliveryFailureContinues: (s)FTP delivery of CDR files failed on retries. BillingServerAddress: blackjack.cs.uoguelph.ca App ID:Cisco CDR Repository Manager Cluster ID: Node ID:iptccm105 



--- 
Lelio Fulgenzi, B.A. 
Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1 
(519) 824-4120 x56354 (519) 767-1060 FAX (ANNU) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
Cooking with unix is easy. You just sed it and forget it. 
- LFJ (with apologies to Mr. Popeil) 



From: "Ryan Ratliff" < rratliff at cisco.com > 
To: "Lelio Fulgenzi" < lelio at uoguelph.ca > 
Cc: cisco-voip at puck.nether.net 
Sent: Friday, February 22, 2013 2:01:13 PM 
Subject: Re: [cisco-voip] Differences between DRS and CDR SFTP activities 



What do the logs for CDR Repository Manager (or CDR Agent, I don't recall offhand which one does the sftp) say about why it's failing? The only immediate difference I can think of is the nature of the files being written though that doesn't really explain why filesystem ownership has an impact. 


-Ryan 


On Feb 22, 2013, at 1:42 PM, Lelio Fulgenzi < lelio at uoguelph.ca > wrote: 



So, we've set up an SFTP server on a linux box and setup a (mounted) filesystem where a user is not the owner, but part of the group that has rw access. WinSCP works fine, user is able to write files. DRS is fine, backups work fine. CDR billing server does not work, transfers fail with partially written files and then others don't go through at all. 

The only way we got CDR transfers to work, was when use used a local filesystem where the user was the owner used a user owned the mounted filesystem it worked. Again, DRS works fine without any of these issues. 

Can anyone speak to the nuances between the DRS SFTP process and the CDR SFTP process and why one would work and one would not? 



--- 
Lelio Fulgenzi, B.A. 
Senior Analyst (CCS) * University of Guelph * Guelph, Ontario N1G 2W1 
(519) 824-4120 x56354 (519) 767-1060 FAX (ANNU) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
Cooking with unix is easy. You just sed it and forget it. 
- LFJ (with apologies to Mr. Popeil) 


_______________________________________________ 
cisco-voip mailing list 
cisco-voip at puck.nether.net 
https://puck.nether.net/mailman/listinfo/cisco-voip 


_______________________________________________ 
cisco-voip mailing list 
cisco-voip at puck.nether.net 
https://puck.nether.net/mailman/listinfo/cisco-voip 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-voip/attachments/20130222/30bbc600/attachment.html>

More information about the cisco-voip mailing list