Skip Menu |

This queue is for tickets about the File-Slurp CPAN distribution.

Report information
The Basics
Id: 38640
Status: resolved
Priority: 0/
Queue: File-Slurp
People
Owner: Nobody in particular
Requestors: cpan [...] pjedwards.co.uk
Cc:
AdminCc:
Bug Information
Severity: (no value)
Broken in: 9999.13
Fixed in: (no value)

History Show all quoted text
  Thu Aug 21 11:22:33 2008 cpan [...] pjedwards.co.uk - Ticket created
Subject: Note: memory not returned to OS.
Hello and thanks for File::Slurp. I was using File::Slurp with some large files and I noticed memory wasn't being returned to the OS when scalars fell out of scope, I thought it worth mentioning, and was thinking it would be useful in the docs. perl -e "use strict;use File::Slurp;sub foo{print system(q{ps aux | grep -E 'BIG|MEM'}).qq{\n}};foo();my \$t=File::Slurp::read_file ('BIG_file.txt');foo();undef(\$t);foo();" %MEM 0.0 %MEM 5.8 %MEM 2.9 Compared to: perl -e "use strict;sub foo{print system(q{ps aux | grep - E 'BIG|MEM'}).qq{\n}};foo();my \$t;{local \$/;open(my \$fh,'BIG_file.txt');\$t=<\$fh>;};foo();undef(\$t);foo();" %MEM 0.0 %MEM 2.9 %MEM 0.0 Cheers, Peter (Stig) Edwards
  Thu Aug 21 11:35:04 2008 cpan [...] pjedwards.co.uk - Correspondence added
CORRECTION, it's not when scalars fell out of scope, but when I call undef() on them.
  Sun Oct 26 14:43:34 2008 DROLSKY [...] cpan.org - Correspondence added
On Thu Aug 21 11:22:33 2008, cpan@pjedwards.co.uk wrote: Show quoted text
> Hello and thanks for File::Slurp. > > I was using File::Slurp with some large files and I noticed memory > wasn't being returned to the OS when scalars fell out of scope, I > thought it worth mentioning, and was thinking it would be useful in the > docs.
This is just how Perl works. It never returns memory that it allocates to the OS. This can make slurping problematic, but it's not specific to this module. Anything that loads a lot of data into memory could be a problem. I suppose a note in the docs wouldn't hurt, but it's not module-specific.
  Sun Oct 26 14:43:40 2008 The RT System itself - Status changed from 'new' to 'open'
  Mon Nov 03 13:05:26 2008 cpan [...] pjedwards.co.uk - Correspondence added
On Sun Oct 26 14:43:34 2008, DROLSKY wrote: Show quoted text
> On Thu Aug 21 11:22:33 2008, cpan@pjedwards.co.uk wrote:
> > Hello and thanks for File::Slurp. > > > > I was using File::Slurp with some large files and I noticed memory > > wasn't being returned to the OS when scalars fell out of scope, I > > thought it worth mentioning, and was thinking it would be useful in
the Show quoted text
> > docs.
> > This is just how Perl works. It never returns memory that it allocates > to the OS. This can make slurping problematic, but it's not specific
to Show quoted text
> this module. Anything that loads a lot of data into memory could be a > problem. > > I suppose a note in the docs wouldn't hurt, but it's not module-
specific. Thanks for the reply, here are some more tests I performed. I thought they might be useful. Using buf_ref avoids the $buf (line 80 of Slurp.pm in File-Slurp- 9999.13) being used, so for example: perl -e "use strict;use File::Slurp;sub foo{print system(q{ps aux | grep -E 'BIG|MEM'}).qq{\n}};foo();my \$t;File::Slurp::read_file ('BIG_file.txt',buf_ref=>\\\$t);foo();undef(\$t);foo();" %MEM 0.0 %MEM 2.9 %MEM 0.0 uses half the memory (as reported by ps %MEM) of $t=File::Slurp::read_file('BIG_file.txt') Also using scalar_ref=>1, uses less memory: perl -e "use strict;use File::Slurp;sub foo{print system(q{ps aux | grep -E 'BIG|MEM'}).qq{\n}};foo();my \$t=File::Slurp::read_file ('BIG_file.txt',scalar_ref=>1);foo();undef(\$t);foo();" %MEM 0.0 %MEM 2.9 %MEM 0.0 Cheers.
  Thu Apr 22 01:53:05 2010 uri [...] stemsystems.com - Correspondence added
CC: cpan [...] pjedwards.co.uk
Subject: Re: [rt.cpan.org #38640] Note: memory not returned to OS.
Date: Thu, 22 Apr 2010 01:52:33 -0400
To: bug-File-Slurp [...] rt.cpan.org
From: "Uri Guttman" <uri [...] StemSystems.com>
Show quoted text
>>>>> "DvR" == DROLSKY via RT <bug-File-Slurp@rt.cpan.org> writes:
Show quoted text
DvR> Queue: File-Slurp DvR> Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=38640 >
Show quoted text
DvR> On Thu Aug 21 11:22:33 2008, cpan@pjedwards.co.uk wrote:
Show quoted text
>> Hello and thanks for File::Slurp. >> >> I was using File::Slurp with some large files and I noticed memory >> wasn't being returned to the OS when scalars fell out of scope, I >> thought it worth mentioning, and was thinking it would be useful in the >> docs.
Show quoted text
DvR> This is just how Perl works. It never returns memory that it allocates DvR> to the OS. This can make slurping problematic, but it's not specific to DvR> this module. Anything that loads a lot of data into memory could be a DvR> problem.
Show quoted text
DvR> I suppose a note in the docs wouldn't hurt, but it's not DvR> module-specific.
i have added notes to the pod for scalar_ref and buf_ref that using them will be faster and save ram usage by not doing an extra copy. i am working (finally) on a new release and catching up on many requests. thanx, uri -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------
  Mon May 23 01:34:26 2011 uri [...] sysarch.com - Status changed from 'open' to 'resolved'