| Subject: | possible memory leak |
| Date: | Tue, 4 Nov 2014 16:17:30 +0000 |
| To: | "bug-Text-CSV_XS [...] rt.cpan.org" <bug-Text-CSV_XS [...] rt.cpan.org> |
| From: | Martin Smoot <MSmoot [...] ejcaimen.com> |
there appears to be a memory leak under Linux when using this code. on 2 32 bit Linux machines (perl V5.12.2) which have either version 0.73 or 0.76 of this package the program runs out of memory if the following sequence is done within a sub when processing (splitting) each line of a number of files (there are over 45000 files totaling around 24 million lines to be split):
my $CSV = Text::CSV_XS->new ({binary => 1, escape_char => "\\"}); # need binary and change escape character
if ($CSV->parse($line_to_split))
if the "my $CSV" line is moved outside the sub then it does not appear to use an abnormal amount of memory. the observed memory usage under Linux is:
27724K - my "outside" sub after processing around 7000 files
307000K - my "inside" sub after processing around 2500 files (and usage is continuing to increase)
it looks like both Linux machines ran out of memory (before the "my" moving fix) at the same time (about 26000 files).
on Windows (package version 0.85, Active State Perl 5.14.2) there does not appear to be a memory problem - reported memory use is stable - maybe up a MB or two - after processing 11000 files.
2-3 years ago when running under Cygwin (32 bit) I had an unknown problem when doing something similar and now suspect that I may have run out of memory. I think that it was Perl 5.10.1 and the package version was 0.76 but could have been an older version of each (I got a new machine). I am not sure if I can run a test to recreate this problem under Cygwin to confirm this.
Show quoted text
________________________________
This message is private and confidential. If you have received this message in error, please notify us by replying to this email and then delete it from your system. Thank you for your cooperation. EJ Caimen, Inc.