Skip Menu |

This queue is for tickets about the Net-SFTP-Foreign CPAN distribution.

Report information
The Basics
Id: 84862
Status: resolved
Priority: 0/
Queue: Net-SFTP-Foreign

People
Owner: Nobody in particular
Requestors: Thomas.Wadley [...] acxiom.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Appending newline character to downloaded file.
Date: Thu, 25 Apr 2013 23:52:56 +0000
To: "bug-Net-SFTP-Foreign [...] rt.cpan.org" <bug-Net-SFTP-Foreign [...] rt.cpan.org>
From: Wadley Thomas - twadle <Thomas.Wadley [...] acxiom.com>
I'm downloading a fixed flat file with no newlines in it, and the file saves with a newline appended. Original size = 19679010 After download = 19679011 I've tried changing loads of options, and haven't found anything that works. $VERSION = '1.62'; perl, v5.8.8 built for x86_64-linux-thread-multi Linux ### 2.6.18-194.11.3.el5 #1 SMP Mon Aug 23 15:51:38 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux What gives? Thomas Wadley Associate Solutions Developer [acxiom] Acxiom Corporation EML thomas.wadley@acxiom.com<mailto:thomas.wadley@acxiom.com> TEL 501.342.7818 301 E. Dave Ward Dr, Conway AR, 72032, USA www.acxiom.com<http://www.acxiom.com/> [Description: Friend Us on Facebook]<http://www.facebook.com/acxiomcorp> [Description: Link Us on LinkedIn] <http://www.linkedin.com/groupRegistration?gid=2901735> [Description: Follow Us on Twitter] <http://twitter.com/acxiom> [Description: cid:image004.png@01CB84F1.26214350] *************************************************************************** The information contained in this communication is confidential, is intended only for the use of the recipient named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please resend this communication to the sender and delete the original message or any copy of it from your computer system. Thank You. ****************************************************************************
Download image001.jpg
image/jpeg 2k
image001.jpg
Download image002.gif
image/gif 1.3k
image002.gif
Download image003.gif
image/gif 1.3k
image003.gif
Download image004.gif
image/gif 1.4k
image004.gif
Download image005.png
image/png 690b
image005.png
Subject: Re: [rt.cpan.org #84862] Appending newline character to downloaded file.
Date: Fri, 26 Apr 2013 09:32:04 +0200
To: bug-Net-SFTP-Foreign [...] rt.cpan.org
From: Salvador Fandiño <sfandino [...] yahoo.com>
On 04/26/2013 01:53 AM, Wadley Thomas - twadle via RT wrote: Show quoted text
> Thu Apr 25 19:53:19 2013: Request 84862 was acted upon. > Transaction: Ticket created by Thomas.Wadley@acxiom.com > Queue: Net-SFTP-Foreign > Subject: Appending newline character to downloaded file. > Broken in: (no value) > Severity: (no value) > Owner: Nobody > Requestors: Thomas.Wadley@acxiom.com > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=84862 > > > > I'm downloading a fixed flat file with no newlines in it, and the file saves with a newline appended. > > Original size = 19679010 > After download = 19679011 > > I've tried changing loads of options, and haven't found anything that works. > $VERSION = '1.62'; > > perl, v5.8.8 built for x86_64-linux-thread-multi > Linux ### 2.6.18-194.11.3.el5 #1 SMP Mon Aug 23 15:51:38 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux > > > What gives?
Without seeing your script source code is hard to tell. Enable debugging on the module adding the following line at the beginning: $Net::SFTP::Foreign::debug = -1; Then show me the first and last thousand lines (note that parts of the content of the transferred file will be included there, if it has sensible information you would like to replace it by a faked one).
Subject: RE: [rt.cpan.org #84862] Appending newline character to downloaded file.
Date: Fri, 26 Apr 2013 16:18:36 +0000
To: "bug-Net-SFTP-Foreign [...] rt.cpan.org" <bug-Net-SFTP-Foreign [...] rt.cpan.org>
From: Wadley Thomas - twadle <Thomas.Wadley [...] acxiom.com>

Message body is not shown because it is too large.

Subject: Re: [rt.cpan.org #84862] Appending newline character to downloaded file.
Date: Fri, 26 Apr 2013 19:27:46 +0200
To: bug-Net-SFTP-Foreign [...] rt.cpan.org
From: Salvador Fandiño <sfandino [...] yahoo.com>
On 04/26/2013 06:19 PM, Wadley Thomas - twadle via RT wrote: Show quoted text
> Queue: Net-SFTP-Foreign > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=84862 > > > The ftp module is tied into an automation agent, which spawns multiple ftp handles and forks, cycling through each one and downloading the detected files for multiple detect locations that have the same ftp site. > > Here's some snippets of the connection and download calls. > > $Net::SFTP::Foreign::debug = -1; > > $args{host} = $host; > $args{user} = $login; > $args{password} = $password unless ($use_secure eq 'K'); > $args{port} = defined($port) ? $port : 22; > $args{timeout} = 300; > $args{more} = '-v'; > $args{late_set_perm} = 1; > $args{block_size} = 65536; > $connection = Net::SFTP::Foreign->new(%args); > $connection->error > and connection_failure($detection_id > ,$login_failure . "[" . $connection->error . "]" > ,$dbh > ,1 > ,undef > ,$connection->status); > > > $ftp_handle->get($source_path > . "/" > . $filename > ,$download_staging_path > . "/" > . $renamed_file > ,callback => $callback) > or ($ftp_handle->get($source_path > . "/" > . $filename > ,$download_staging_path > . "/" > . $renamed_file > ,callback => $callback) > or download_failure($filename > ,$download_staging_path > ,$renamed_file > ,\%det > ,$file_prelog_id > ,$dbh) > and next); > my callback hook.. > > > sub sftp_callback { > my ($sftp, $data, $offset, $size) = @_; > > my $gigabytes = sprintf("%.4f", $size/1024/1024/1024); > my $block_size = $sftp->{_block_size}; > > if ($size eq 0) { > ## avoid division by zero > $size = 1; > } > ## total full blocks > my $total_blocks = int($size/$block_size); > > if ($offset eq 0) { > vprint ("Starting transfer of $gigabytes GB file"); > vprint ("Block Size = $block_size" ); > > } > my $dyn_percent= sprintf("%.1f",$offset/$size); > > my $print_flag = undef; > > my $calc_offset = int($total_blocks*$dyn_percent+.1)*$block_size; > $print_flag = 1 if ($offset eq $calc_offset); > > if ($offset eq $size || defined($print_flag) ) { > my $percent = sprintf("%.4f",$offset / $size)*100; > > vprint("Transferred [$percent\%] -- [$offset bytes] "); > > } > }
I bet that $\ is being changed to "\n" from the vprint function being invoked from your callback and that, is interfering with Net::SFTP::Foreign get method which uses "print" to write the file chunks to the destination file under the hood. If that is the case, use local to limit the scope of the $\ change to the vprint function. In example: sub vprint { local $\; ... In any case, I am also going to add some defensive code around the callback invocations in Net::SFTP::Foreign to protect against this issue.
Subject: Re: [rt.cpan.org #84862] Appending newline character to downloaded file.
Date: Fri, 26 Apr 2013 19:52:55 +0200
To: bug-Net-SFTP-Foreign [...] rt.cpan.org
From: Salvador Fandiño <sfandino [...] yahoo.com>
On 04/26/2013 07:29 PM, Salvador \"Fandiño\" via RT wrote: Show quoted text
> Queue: Net-SFTP-Foreign > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=84862 > > > On 04/26/2013 06:19 PM, Wadley Thomas - twadle via RT wrote:
>> Queue: Net-SFTP-Foreign >> Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=84862 > >> >> The ftp module is tied into an automation agent, which spawns multiple ftp handles and forks, cycling through each one and downloading the detected files for multiple detect locations that have the same ftp site. >> >> Here's some snippets of the connection and download calls. >> >> $Net::SFTP::Foreign::debug = -1; >> >> $args{host} = $host; >> $args{user} = $login; >> $args{password} = $password unless ($use_secure eq 'K'); >> $args{port} = defined($port) ? $port : 22; >> $args{timeout} = 300; >> $args{more} = '-v'; >> $args{late_set_perm} = 1; >> $args{block_size} = 65536; >> $connection = Net::SFTP::Foreign->new(%args); >> $connection->error >> and connection_failure($detection_id >> ,$login_failure . "[" . $connection->error . "]" >> ,$dbh >> ,1 >> ,undef >> ,$connection->status); >> >> >> $ftp_handle->get($source_path >> . "/" >> . $filename >> ,$download_staging_path >> . "/" >> . $renamed_file >> ,callback => $callback) >> or ($ftp_handle->get($source_path >> . "/" >> . $filename >> ,$download_staging_path >> . "/" >> . $renamed_file >> ,callback => $callback) >> or download_failure($filename >> ,$download_staging_path >> ,$renamed_file >> ,\%det >> ,$file_prelog_id >> ,$dbh) >> and next); >> my callback hook.. >> >> >> sub sftp_callback { >> my ($sftp, $data, $offset, $size) = @_; >> >> my $gigabytes = sprintf("%.4f", $size/1024/1024/1024); >> my $block_size = $sftp->{_block_size}; >> >> if ($size eq 0) { >> ## avoid division by zero >> $size = 1; >> } >> ## total full blocks >> my $total_blocks = int($size/$block_size); >> >> if ($offset eq 0) { >> vprint ("Starting transfer of $gigabytes GB file"); >> vprint ("Block Size = $block_size" ); >> >> } >> my $dyn_percent= sprintf("%.1f",$offset/$size); >> >> my $print_flag = undef; >> >> my $calc_offset = int($total_blocks*$dyn_percent+.1)*$block_size; >> $print_flag = 1 if ($offset eq $calc_offset); >> >> if ($offset eq $size || defined($print_flag) ) { >> my $percent = sprintf("%.4f",$offset / $size)*100; >> >> vprint("Transferred [$percent\%] -- [$offset bytes] "); >> >> } >> }
> I bet that $\ is being changed to "\n" from the vprint function being > invoked from your callback and that, is interfering with > Net::SFTP::Foreign get method which uses "print" to write the file > chunks to the destination file under the hood. > > If that is the case, use local to limit the scope of the $\ change to > the vprint function. In example: > > sub vprint { > local $\; > ... > > In any case, I am also going to add some defensive code around the > callback invocations in Net::SFTP::Foreign to protect against this issue.
I have just uploaded 1.76_01 to CPAN: https://metacpan.org/release/SALVA/Net-SFTP-Foreign-1.76_01
No feedback from the reporter. I am supposing the patch solved the problem and closing the ticket.
Subject: RE: [rt.cpan.org #84862] Resolved: Appending newline character to downloaded file.
Date: Fri, 24 May 2013 14:26:59 +0000
To: "bug-Net-SFTP-Foreign [...] rt.cpan.org" <bug-Net-SFTP-Foreign [...] rt.cpan.org>
From: Wadley Thomas - twadle <Thomas.Wadley [...] acxiom.com>
Thank you! Show quoted text
-----Original Message----- From: Salvador Fandino Garcia via RT [mailto:bug-Net-SFTP-Foreign@rt.cpan.org] Sent: Friday, May 24, 2013 3:30 AM To: Wadley Thomas - twadle Subject: [rt.cpan.org #84862] Resolved: Appending newline character to downloaded file. <URL: https://rt.cpan.org/Ticket/Display.html?id=84862 > According to our records, your request has been resolved. If you have any further questions or concerns, please respond to this message. *************************************************************************** The information contained in this communication is confidential, is intended only for the use of the recipient named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please resend this communication to the sender and delete the original message or any copy of it from your computer system. Thank You. ****************************************************************************