Bug #24072 for Tie-File: Document in-memory support

Sun Dec 24 15:32:36 2006 mschwern [...] cpan.org - Ticket created

Subject:

Document in-memory support

Tie::File can do in-memory access, if you really want it to. open FH, ">", \$scalar; tie @rec, 'Tie::File', \*FH; The same caveats apply for this filehandle as any other. It would be nice to mention this somewhere. It should also mention that its memory efficient for large arrays with small elements (ie. a huge list of integers) but its performance inefficient, O(n**2) and that there are better ways to efficiently store large lists of small elements in memory. Tie::VecArray, PDL (probably has tied array access somewhere), Bit::Vector::Array and DB_File.

Sun Dec 24 15:33:22 2006 mschwern [...] cpan.org - Requestor dankogai@dan.co.jp added

Sun Dec 24 15:34:04 2006 mschwern [...] cpan.org - Requestor dankogai@dan.co.jp deleted

Sun Dec 24 15:34:05 2006 mschwern [...] cpan.org - Requestor DANKOGAI added

Sun Dec 24 18:34:04 2006 mjd [...] plover.com - Correspondence added

Subject:	Re: [rt.cpan.org #24072] Document in-memory support
Date:	Sun, 24 Dec 2006 18:33:51 -0500
To:	bug-Tie-File [...] rt.cpan.org
From:	Mark Jason Dominus <mjd [...] plover.com>

Show quoted text

> Tie::File can do in-memory access, if you really want it to. > > open FH, ">", \$scalar; > tie @rec, 'Tie::File', \*FH;

It's already documented that you can supply a pre-opened filehandle. There is a whole section of the manual titled "Tying to an already-opened filehandle". I have attached it below for your convenience. I question the usefulnes of adding any material about in-memory filehandles. ---------------------------------------------------------------- Tying to an already-opened filehandle If $fh is a filehandle, such as is returned by "IO::File" or one of the other "IO" modules, you may use: tie @array, 'Tie::File', $fh, ...; Similarly if you opened that handle "FH" with regular "open" or "sysopen", you may use: tie @array, 'Tie::File', \*FH, ...; Handles that were opened write-only won't work. Handles that were opened read-only will work as long as you don't try to modify the array. Handles must be attached to seekable sources of data---that means no pipes or sockets. If "Tie::File" can detect that you supplied a non-seekable handle, the "tie" call will throw an exception. (On Unix systems, it can detect this.) Note that Tie::File will only close any filehandles that it opened internally. If you passed it a filehandle as above, you "own" the filehandle, and are responsible for closing it after you have untied the @array.

Sun Dec 24 18:34:06 2006 The RT System itself - Status changed from 'new' to 'open'

Mon Dec 25 01:01:03 2006 schwern [...] pobox.com - Correspondence added

CC:	DANKOGAI [...] cpan.org, MSCHWERN [...] cpan.org
Subject:	Re: [rt.cpan.org #24072] Document in-memory support
Date:	Mon, 25 Dec 2006 01:00:34 -0500
To:	bug-Tie-File [...] rt.cpan.org
From:	Michael G Schwern <schwern [...] gmail.com>

Mark-Jason Dominus via RT wrote: Show quoted text

> I question the usefulnes of adding any material about in-memory > filehandles.

Sometimes folks need a little nudge to make a seemingly simple mental leap to remember A) about in-memory filehandles, B) that Tie::File takes filehandles and C) to put A and B together. I'd forgotten about in-memory filehandles. Dan remembered them but assumed it just wouldn't work. I searched for "memory" and didn't find anything relevant so I just figured it couldn't be done. Something as simple as this is that nudge. It also provides the memory/speed tradeoff rationale. If you think the rationale is a silly one and they just shouldn't be used with Tie::File then yes, just leave out mention of them. =head1 Tying to an in-memory filehandle You can use an in-memory filehandle with Tie::File same as any other already-opened filehandle. All the same caveats apply. open my $fh, '>', \$stuff; tie @array, "Tie::File", $fh, ...; Why would you do this rather than simply use an untied array? Regular arrays have a significant memory overhead per element. If you need to store a large array with small elements, say a large list of numbers, this will consume less memory as they are stored in a string. However, access will be significantly slower than a normal array.

Mon Dec 25 11:01:17 2006 mjd [...] plover.com - Correspondence added

Subject:	Re: [rt.cpan.org #24072] Document in-memory support
Date:	Mon, 25 Dec 2006 11:00:59 -0500
To:	bug-Tie-File [...] rt.cpan.org
From:	Mark Jason Dominus <mjd [...] plover.com>

Okay, but as yet nobody has provided any explanantion of why this is worth doing. Show quoted text

> Why would you do this rather than simply use an untied array? > Regular arrays have a significant memory overhead per element. If > you need to store a large array with small elements, say a large > list of numbers, this will consume less memory

This is pure fantasy; you pulled it right out of your ass. As I said earlier, Tie::File is not going to save any memory here. Consider the particular case of a list of numbers. Okay, the numbers are stored in a string. Great. But, in order to record the position at which each of the stored numbers occurs in the string, Tie::File uses...an array of numbers. Let's drop this thread until you or Dan or someone can come up with a plausible explanantion of why this would be worth doing. If the explanation is based on performance, it must be accompanied by some evidence. So far, the whole discussion seems ill-conceived, based on mistaken assumptions insufficiently investigated. I started out by supposing that there was something going on that I was not understanding, but the longer the conversation continues, the more I lose hope that that is the case. Happy New Year.

Mon Dec 25 17:06:24 2006 schwern [...] pobox.com - Correspondence added

CC:	DANKOGAI [...] cpan.org, MSCHWERN [...] cpan.org
Subject:	Re: [rt.cpan.org #24072] Document in-memory support
Date:	Mon, 25 Dec 2006 17:06:02 -0500
To:	bug-Tie-File [...] rt.cpan.org
From:	Michael G Schwern <schwern [...] gmail.com>

Mark-Jason Dominus via RT wrote: Show quoted text

> <URL: http://rt.cpan.org/Ticket/Display.html?id=24072 > > > > Okay, but as yet nobody has provided any explanantion of why this is > worth doing. >

>> Why would you do this rather than simply use an untied array? >> Regular arrays have a significant memory overhead per element. If >> you need to store a large array with small elements, say a large >> list of numbers, this will consume less memory

> > This is pure fantasy; you pulled it right out of your ass.

Well, Dan's ass which is significantly kinkier. :) Show quoted text

> As I said earlier, Tie::File is not going to save any memory here. > > Consider the particular case of a list of numbers. Okay, the numbers > are stored in a string. Great. But, in order to record the position > at which each of the stored numbers occurs in the string, Tie::File > uses...an array of numbers. > > Let's drop this thread until you or Dan or someone can come up with a > plausible explanantion of why this would be worth doing. If the > explanation is based on performance, it must be accompanied by some > evidence. > > So far, the whole discussion seems ill-conceived, based on mistaken > assumptions insufficiently investigated. I started out by supposing > that there was something going on that I was not understanding, but > the longer the conversation continues, the more I lose hope that that > is the case.

You're right, I reran some benchmarks and didn't see any memory savings.

Sat Jan 18 22:51:59 2020 TODDR [...] cpan.org - Correspondence added

Ticket migrated to github as https://github.com/toddr/Tie-File/issues/11

Sat Jan 18 22:52:00 2020 TODDR [...] cpan.org - Status changed from 'open' to 'resolved'