| Subject: | Memory leaking |
| Date: | Thu, 29 Nov 2012 15:21:46 +0100 |
| To: | bug-Text-CSV_XS [...] rt.cpan.org |
| From: | Matthieu Maury <mayeu.tik [...] gmail.com> |
Hi,
At my company we work on a software which handle a lot of different
type of input file.
The software is a long lasting software running on a server.
We recently had the need to add the possibility to read data from CSV
file and we add a little piece of driver using Text::CSV_XS (we need
speed, a lot of speed).
But using Text::CSV_XS 0.93 made you software blow up under ram
leaking (the script was taking something like 10Gb of memory, but
usualy the script run on 250Mb ~ 500Mb of memory).
We create a little script presenting the issue :
------------------------------
--------------------
use Text::CSV;
$i = 0;
foreach $file (@ARGV) {
open $FILE, "<$file";
my $CSVFile = Text::CSV->new(
{ 'quote_char' => '"',
'sep_char' => ';',
'always_quote' => 0,
'quote_space' => 1,
'quote_null' => 1,
'binary' => 1,
'blank_is_undef' => 1,
'empty_is_undef' => 1,
}
);
my $header_ref = $CSVFile->getline($FILE);
my $columns_ref = $CSVFile->getline($FILE);
$CSVFile->column_names( @{$columns_ref} );
my $lines = $CSVFile->getline_hr_all($FILE);
close FILE ;
print "File $i done\n" ;
$i++ ;
}
---------------------------------------------------
By calling this script with 200 csv file in input (and with
Text::CSV_XS installed), my perl process increase its memory at every
loop, if I remove Text::CSV_XS (and thus using the pure perl
implementation of Text::CSV), my perl process use a fixed quantity of
memory.
We got this issue on different system :
Archlinux, perl v5.16.2 x86_64-linux-thread-multi (from arch
repository), Text::CSV_XS 0.93
Debian Sqeeze, v5.10. x86_64-linux-gnu-thread-multi (from debian
repository, TexT::CSV_XS from debian repository (libtext-csv-xs-perl).
Thanks for your help and insight.
--
Mayeu
http://6x9.fr