Bug #43156 for File-Slurp: Slurp not handling 4GB+ files

Mon Feb 09 15:56:37 2009 kissifrot [...] gmail.com - Ticket created

Subject:	Slurp not handling 4GB+ files
Date:	Mon, 9 Feb 2009 21:56:19 +0100
To:	bug-File-Slurp [...] rt.cpan.org
From:	Philippe Villiers <kissifrot [...] gmail.com>

Hello, I'm trying to use File::Slurp (version 9999.13) to open big files (4GB+), and I get the following error: Negative length at /usr/lib/perl5/vendor_perl/5.8.8/File/Slurp.pm line 159 Perl is compiled with large file support however: Characteristics of this binary (from libperl): Compile-time options: PERL_MALLOC_WRAP USE_LARGE_FILES USE_PERLIO Built under linux Compiled at Dec 15 2008 09:53:57 @INC: /etc/perl /usr/lib/perl5/vendor_perl/5.8.8/i686-linux /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib/perl5/site_perl/5.8.8/i686-linux /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib/perl5/5.8.8/i686-linux /usr/lib/perl5/5.8.8 /usr/local/lib/site_perl . Do you have any idea on how to solve my problem? Regards, -- Philippe Villiers

Fri Nov 19 01:10:21 2010 sharyanto [...] cpan.org - Correspondence added

On Mon Feb 09 15:56:37 2009, kissifrot@gmail.com wrote: Show quoted text

> Hello, > > I'm trying to use File::Slurp (version 9999.13) to open big files > (4GB+), and I get the following error: > Negative length at /usr/lib/perl5/vendor_perl/5.8.8/File/Slurp.pm

line 159 Show quoted text

> > Perl is compiled with large file support however: > Characteristics of this binary (from libperl): > Compile-time options: PERL_MALLOC_WRAP USE_LARGE_FILES USE_PERLIO > Built under linux > Compiled at Dec 15 2008 09:53:57 > @INC: > /etc/perl > /usr/lib/perl5/vendor_perl/5.8.8/i686-linux > /usr/lib/perl5/vendor_perl/5.8.8 > /usr/lib/perl5/vendor_perl > /usr/lib/perl5/site_perl/5.8.8/i686-linux > /usr/lib/perl5/site_perl/5.8.8 > /usr/lib/perl5/site_perl > /usr/lib/perl5/5.8.8/i686-linux > /usr/lib/perl5/5.8.8 > /usr/local/lib/site_perl > . > > Do you have any idea on how to solve my problem? > > Regards,

Are you on a 32bit system? If so, you're probably affected by the 4GB max process size on 32bit systems, since the whole file is taken into memory (into the Perl variable). Try doing it on a 64bit system. -- sh

Fri Nov 19 01:10:23 2010 The RT System itself - Status changed from 'new' to 'open'

Fri Nov 19 01:20:38 2010 uri [...] stemsystems.com - Correspondence added

Subject:	Re: [rt.cpan.org #43156] Slurp not handling 4GB+ files
Date:	Fri, 19 Nov 2010 01:20:24 -0500
To:	bug-File-Slurp [...] rt.cpan.org
From:	"Uri Guttman" <uri [...] StemSystems.com>

Show quoted text

>>>>> "shvR" == steven haryanto via RT <bug-File-Slurp@rt.cpan.org> writes:

Show quoted text

shvR> Are you on a 32bit system? If so, you're probably affected by the 4GB shvR> max process size on 32bit systems, since the whole file is taken into shvR> memory (into the Perl variable). Try doing it on a 64bit system.

good point. also you can be on a 32 bit cpu but have 64 bit file seek values. slurp can't handle that as it uses perl's scalars for its calculations. but you (the OP) shouldn't be slurping in files that large. in my article (see the extras/) i say slurping should be normal for many files which used to be too large such as 10-100 megabyte files. hell, years ago 32k files were deemed large. that is why line by line i/o was invented and used so much. uri file::slurp author -- Uri Guttman ------ uri@stemsystems.com -------- http://www.sysarch.com -- ----- Perl Code Review , Architecture, Development, Training, Support ------ --------- Gourmet Hot Cocoa Mix ---- http://bestfriendscocoa.com ---------

Mon May 23 01:19:14 2011 uri [...] sysarch.com - Correspondence added

not a problem with File::Slurp as you shouldn't be slurping in files that large.

Mon May 23 01:19:15 2011 uri [...] sysarch.com - Status changed from 'open' to 'rejected'