Skip Menu |

This queue is for tickets about the File-Map CPAN distribution.

Report information
The Basics
Id: 56134
Status: resolved
Priority: 0/
Queue: File-Map

People
Owner: LEONT [...] cpan.org
Requestors: user42 [...] zip.com.au
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: (no value)
Fixed in: 0.35



Subject: check handle layers before mapping
Date: Wed, 31 Mar 2010 11:04:07 +1100
To: bug-File-Map [...] rt.cpan.org
From: Kevin Ryde <user42 [...] zip.com.au>
In my locate iterator, when given a file handle I look at the perlio layers as part of choosing mmap or plain reads. Any layers that might transform the data would be bypassed by an mmap so plain reads must be used. As an idea for a feature, File::Map could have something to say whether map_handle() will give the same bytes as plain reads. Maybe map_handle() could go so far as to refuse to operate on a handle with transforms and/or utf-8 flag. Refusing would avoid surprises seeing data different than read(), but might be inconvenient for anyone who thinks they know what they're doing. In my code I list some known-safe layers like :unix and :stdio and rate anything unknown as possibly transforming the data and therefore unsafe. A more general mechanism might be to look for a class method like PerlIO::foo->File_Map_clean which could say if bypassing the :foo layer by an mmap is ok. (Keeping the linkage loose, so File::Map doesn't have to hard code a list, and layers don't have to care if File::Map is ever loaded or used.) On ms-dos I think the C library unix emulations add an O_TEXT or some such flag on file descriptors. Dunno if an mmap() in that case obeys that flag or skips it. If skipped then the same consideration would apply there as for :crlf pushed on a perl-level handle.
I've fixed this in HEAD. I'm using the same safelist you did. I'm not so sure about the value of calling method on Perlio::* as I can't think of any use case where that would make sense. So until there an actual need for it I'll refrain from adding it. I could theoretically support :utf8 but knowing the disadvantages of that compared to :encoding(utf8) I don't think that's good idea. I should probably add something about that in the pitfalls section though.
Subject: Re: [rt.cpan.org #56134] check handle layers before mapping
Date: Sat, 03 Apr 2010 10:56:46 +1100
To: bug-File-Map [...] rt.cpan.org
From: Kevin Ryde <user42 [...] zip.com.au>
"Leon Timmermans via RT" <bug-File-Map@rt.cpan.org> writes: Show quoted text
> > I've fixed this in HEAD. I'm using the same safelist you did.
Beaut. Show quoted text
> I'm not so > sure about the value of calling method on Perlio::* as I can't think of > any use case where that would make sense.
Oh well, you know you don't want to have to hard-code knowledge about other modules and layers, or be obliged to update when someone's snazzy new multi-chunk-buffering layer comes along, etc. If there was something in the builtin :perlio etc layers which identified them as straight-through then of course it'd be good not even to list them explicitly. The main things against a method call-out that I can see would be that PerlIO::foo may not always an actual class (though that may not matter). And ideally you'd like to tell it the particular instance of the layer that you're asking about, in case mmap-cleanness depended on some flags in it or something. I'm not sure there is such a layer instance object, except perhaps in PerlIO::via, and even then I wouldn't know how to access it. Show quoted text
> :utf8
I was going to start a ticket for wondering what happens if you turn on the utf8 flag of an mmapped scalar, or whether that flag is copied across if there's :utf8 on the handle that you map. My guess would be utf8 on the write side would bomb badly if the encoding resulted in more bytes, but that the read side might work. I heard a rumour perl can crash if there's invalid utf8 in a variable flagged utf8, or maybe that was only early 5.6 or something. Hope it's not true, if it was then bad file contents might be fatal, which would curtail usefulness a bit.
I'm reopening this ticket. It turns out that there is no reliable way to check if crlf line-end translation is happening or not, resulting in map_handle breaking completely on Win32 for versions 0.24-0.26. There is no way to check this from Perl-land. I'm allowing the crlf layer until I've been able to produce such a module.
Subject: Re: [rt.cpan.org #56134] check handle layers before mapping
Date: Sat, 08 May 2010 11:38:42 +1000
To: bug-File-Map [...] rt.cpan.org
From: Kevin Ryde <user42 [...] zip.com.au>
"Leon Timmermans via RT" <bug-File-Map@rt.cpan.org> writes: Show quoted text
> > I'm reopening this ticket. It turns out that there is no reliable way to > check if crlf line-end translation is happening or not,
crlf shows up on msdos even when there's no translation? Doesn't binmode() strip that off? Or it makes it binary but still shows up in get_layers() or something silly ... Incidentally in 0.27 your link L<binmode> might be meant to be L<perlfunc/binmode>, and space instead of underscore in L<utf8::decode|utf8/"Utility_functions">. (Spotted by my secret new podlinkcheck program :-).
On Fri May 07 21:39:37 2010, user42@zip.com.au wrote: Show quoted text
> crlf shows up on msdos even when there's no translation? Doesn't > binmode() strip that off? Or it makes it binary but still shows up in > get_layers() or something silly ...
It's the latter. It disables the PERLIO_F_CRLF flag, but it doesn't remove the crlf layer (because it also houses the buffering and such). To make matters worse the PerlIO flag definitions aren't available in PerlIO.pm, so a small XS module is required just for that. I'm actually considering sending a patch to p5p, though I'm not sure they would want to dual-life PerlIO.pm. Show quoted text
> Incidentally in 0.27 your link L<binmode> might be meant to be > L<perlfunc/binmode>, and space instead of underscore in > L<utf8::decode|utf8/"Utility_functions">. (Spotted by my secret new > podlinkcheck program :-).
Thanks, I'll update it when I make my next release. Did you consider Making a Test::* module out of it? Leon
Subject: Re: [rt.cpan.org #56134] check handle layers before mapping
Date: Sun, 09 May 2010 09:03:58 +1000
To: bug-File-Map [...] rt.cpan.org
From: Kevin Ryde <user42 [...] zip.com.au>
"Leon Timmermans via RT" <bug-File-Map@rt.cpan.org> writes: Show quoted text
> > doesn't > remove the crlf layer (because it also houses the buffering and such).
Leaving the name behind just to confuse :) Show quoted text
> To make matters worse the PerlIO flag definitions aren't available in > PerlIO.pm, so a small XS module is required just for that.
You could make a private newCONST('File::Map::_PERLIO_F_CRLF') just to make it available I suppose. Show quoted text
> Did you consider > Making a Test::* module out of it?
That's likely, though it might have to be careful about what's considered an error. A missing target might just be something not on cpan any more. If it was recommended only as an "author" test then a few false reports wouldn't matter (esp if particular ones could be silenced somehow).
The bug is finally fixed in File::Map 0.35 :-)