Bug #3222 for Tie-File: using Tie::File

Tue Aug 12 14:58:50 2003 pprasadb [...] planet-save.com - Ticket created

Date:	Mon, 28 Jul 2003 12:24:10 -0700 (PDT)
From:	Prasad Poruporuthan <pprasadb [...] planet-save.com>
To:	mjd-perl-tiefile+ [...] plover.com
Subject:	using Tie::File

Hi mjd, I have written a Parser module and I am trying to use your Tie::File to handle large files... I was trying out some code like this, ======================================================= use Tie::File; use Devel::Size qw(total_size); my @array; tie @array, 'Tie::File', $ARGV[0], memory => 20000000, recsep => '~'; $size = total_size (\@array); print "diagnostics : total_size(\@array) = $size\n"; ======================================================= However, I found that even though I defined max cache use to 20,000,000, the Tie::File module was using a lot more memory. If I specify files as large 50 MB, it overflows my 256MB RAM, and even my 512 MB swap. Could u let me know, if I am doing something wrong. Your help would be appreciated. Thanks, Prasad Show quoted text

_____________________________________________________________ Save rainforest for free with a Planet-Save.com e-mail account: http://www.planet-save.com

Tue Aug 12 15:57:04 2003 mjd [...] plover.com - Correspondence added

To:	bug-Tie-File [...] rt.cpan.org
Subject:	Re: [cpan #3222] using Tie::File
Date:	Tue, 12 Aug 2003 15:57:41 -0400
From:	Mark Jason Dominus <mjd [...] plover.com>
RT-Send-Cc:

Show quoted text

> However, I found that even though I defined max cache use to 20,000,000, > the Tie::File module was using a lot more memory. > > If I specify files as large 50 MB, it overflows my 256MB RAM, > and even my 512 MB swap. > > Could u let me know, if I am doing something wrong.

I'm not sure, but my guess is that it is a problem with the module, not with your code. The manual says: The "memory" value is not an absolute or exact limit on the memory used. "Tie::File" objects contains some structures besides the read cache and the deferred write buffer, whose sizes are not charged against "memory". If your file has many lines, it's possible that this overhead is very large. Can you please tell me how many lines are in your 50MB file? I should probably add that whatever you're trying to use Devel::Size for here, it probably won't work. Using Devel::Size on a tied array only tells you how much space the array is actually using. Since none of the file data or anything else is stored in the array, Devel::Size will report a very small size. If you want to know how much space Tie::File is using, you probably should do total_size(tied @array); instead.

Tue Aug 12 16:03:05 2003 MJD [...] cpan.org - TimeWorked changed from (no value) to '45'

Tue Aug 12 16:03:05 2003 MJD [...] cpan.org - Severity Important added

Tue Aug 12 16:03:05 2003 MJD [...] cpan.org - Status changed from 'new' to 'open'

Tue Aug 12 16:03:06 2003 MJD [...] cpan.org - Broken in 0.93 added

Tue Aug 12 17:14:44 2003 mjd [...] plover.com - Correspondence added

To:	bug-Tie-File [...] rt.cpan.org
Subject:	Re: [cpan #3222] using Tie::File
Date:	Tue, 12 Aug 2003 17:15:21 -0400
From:	Mark Jason Dominus <mjd [...] plover.com>
RT-Send-Cc:

Show quoted text

> ======================================================= > use Tie::File; > use Devel::Size qw(total_size); > > my @array; > tie @array, 'Tie::File', $ARGV[0], memory => 20000000, recsep => '~'; > > $size = total_size (\@array); > print "diagnostics : total_size(\@array) = $size\n"; > ======================================================= > > However, I found that even though I defined max cache use to 20,000,000, > the Tie::File module was using a lot more memory. > > If I specify files as large 50 MB, it overflows my 256MB RAM, > and even my 512 MB swap.

It turns out that in your program above, Devel::Size wil load the entire file into memory. It doesn't surprise me that this program runs out of memory, but this is not a problem with Tie::File; it's a problem with the way Devel::Size is designed, or with the way you are using it. Do you also have problems when you use Tie::File without Devel::Size?

Wed Aug 20 17:28:01 2003 Guest - Correspondence added

From:

Prasad

Mark, I still have problems even if don't use the Devel::Size module. I am using a file with Size 200 MB approx Lines 10,000,000 approx each line length varies between 10 to 100 characters. code executed is use Tie::File; tie @array, 'Tie::File', $ARGV[0], memory => 20000000; $size = @array; Memory usage is about 230 MB. (130 MB Main memory and 100 MB swap). From what you pointed out, even though I have defined memory => 20MB, the Tie::File would use more memory for maintaining its internal structures. But here it is using 250 MB, which seems to be very large.So specifying the memory => 20MB is not really limiting the memory that the Tie::File is using. Is the additional memory that Tie::File dependant on the File size ? Is this behaviour expected ? If I use files > 500MB, the memory usage will still be more. What file sizes can Tie::File use efficiently ? Thanks Prasad

Sat Jan 18 22:51:48 2020 TODDR [...] cpan.org - Correspondence added

Ticket migrated to github as https://github.com/toddr/Tie-File/issues/9

Sat Jan 18 22:51:49 2020 TODDR [...] cpan.org - Status changed from 'open' to 'resolved'