Bug #73872 for File-Map: Can't map with nonzero $offset under Windows

Tue Jan 10 05:56:11 2012 Ralf.Neubauer [...] wido.bv.aok.de - Ticket created

Subject:	Can't map with nonzero $offset under Windows
Date:	Tue, 10 Jan 2012 11:55:58 +0100
To:	<bug-File-Map [...] rt.cpan.org>
From:	"Neubauer, Ralf" <Ralf.Neubauer [...] wido.bv.aok.de>

File::Map 0.40, ActivePerl 5.14.2 build 1402 use File::Map qw(:map :extra :constants); map_file my $map, '/some/big/file', '<', 0x10000, 0x20000; doesn't work under Windows. The same happens with sys_map my $map2, 0x10000, PROT_READ, MAP_FILE, $fh2, 0x10000; while map_file my $map, '/some/big/file', '<', 0x00000, 0x20000; and map_file my $map, '/some/big/file', '<', 0x00000, 0x30000; and sys_map my $map2, 0x10000, PROT_READ, MAP_FILE, $fh2, 0; work. I assume this has something to do with the #ifdef WIN32 implementation of do_mapping() in Map.xs. The two interesting calls are: ... mapping = CreateFileMapping(file, NULL, winflags[prot].createflag, 0, length, NULL); ... address = MapViewOfFile(mapping, viewflag, 0, offset, length); ... According to the documentation of CreateFileMapping passing /length+offset/ instead of /length/ should be correct, else the mapping only maps the _first_ /length/ bytes of file, which doesn't include all of the /length/ bytes from /offset/. Also the /0/ arguments of CreateFileMapping and MapViewOfFile take the high 32 bits of the length resp. the offset. If off_t wouldn't be long (32 bits) anyway (or Map.xs used another type), it would be easy to support large files under win32. -- Ralf Neubauer Wissenschaftliches Institut der AOK (WIdO) Rosenthaler Straße 31 10178 Berlin Tel: 030 34646-2583 Fax: 030 34646-2144 E-Mail: Ralf.Neubauer@wido.bv.aok.de

Sat Jan 14 07:37:13 2012 LEONT [...] cpan.org - Correspondence added

On Tue Jan 10 05:56:11 2012, Ralf.Neubauer@wido.bv.aok.de wrote: Show quoted text

> According to the documentation of CreateFileMapping passing > /length+offset/ instead of /length/ should be correct, else the > mapping only maps the _first_ /length/ bytes of file, which doesn't > include all of the /length/ bytes from /offset/.

Thanks for the clear bug report. I just released 0.41, which contains a fix for this (as well as an other, minor Win32 issue that I spotted along the way). Show quoted text

> Also the /0/ arguments of CreateFileMapping and MapViewOfFile take the > high 32 bits of the length resp. the offset. If off_t wouldn't be > long (32 bits) anyway (or Map.xs used another type), it would be > easy to support large files under win32.

I'm not sure what type would be appropriate, or how perl on 32bit Windows is usually compiled (wrt 64 bit integer support). Without the latter it's a fairly moot point anyway. Leon

Sat Jan 14 07:37:19 2012 The RT System itself - Status changed from 'new' to 'open'

Sat Jan 14 08:08:09 2012 LEONT [...] cpan.org - Status changed from 'open' to 'resolved'

Mon Jan 16 04:52:23 2012 Ralf.Neubauer [...] wido.bv.aok.de - Correspondence added

Subject:	RE: [rt.cpan.org #73872] Can't map with nonzero $offset under Windows
Date:	Mon, 16 Jan 2012 10:52:10 +0100
To:	<bug-File-Map [...] rt.cpan.org>
From:	"Neubauer, Ralf" <Ralf.Neubauer [...] wido.bv.aok.de>

Show quoted text

> -----Original Message----- > From: Leon Timmermans via RT [mailto:bug-File-Map@rt.cpan.org] > Sent: Saturday, January 14, 2012 1:37 PM >

> > Also the /0/ arguments of CreateFileMapping and MapViewOfFile take the > > high 32 bits of the length resp. the offset. If off_t wouldn't be > > long (32 bits) anyway (or Map.xs used another type), it would be > > easy to support large files under win32.

> > I'm not sure what type would be appropriate, or how perl on 32bit > Windows is usually compiled (wrt 64 bit integer support). Without the > latter it's a fairly moot point anyway.

perl -V says: intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8 alignbytes=8, prototype=define (a 64 bit perl says: intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='__int64', ivsize=8, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8 alignbytes=8, prototype=define ) As I understand that, integer scalars are 32 bits, but perl itself supports 64 bit offsets in files. Which is not a problem, since floating point scalars are of type double and have a 53 bit mantissa, which even with rounding should be ok for 40 or even 48 bit file offsets. Also the Windows-64 Version of MapViewOfFile has exactly the same DWORD arguments for the offset (i.e. 32 bits), so bits must be shifted anyway to pass a 64 bit offset -- the same with the length arguments of CreateFileMapping. With MapViewOfFileEx you can at least specify the length as SIZE_T, which is 64 bits on Windows-64. Ralf

Mon Jan 16 04:52:24 2012 The RT System itself - Status changed from 'resolved' to 'open'

Tue Jan 24 17:26:40 2012 LEONT [...] cpan.org - Correspondence added

On Mon Jan 16 04:52:23 2012, Ralf.Neubauer@wido.bv.aok.de wrote: Show quoted text

> > -----Original Message----- > > From: Leon Timmermans via RT

> [mailto:bug-File-Map@rt.cpan.org]

> > Sent: Saturday, January 14,

> 2012 1:37 PM

> >

> > > Also the /0/ arguments of CreateFileMapping

> and MapViewOfFile take the

> > > high 32 bits of the length resp.

> the offset. If off_t wouldn't be

> > > long (32 bits) anyway (or

> Map.xs used another type), it would be

> > > easy to support

> large files under win32.

> > > > I'm not sure what type would be

> appropriate, or how perl on 32bit

> > Windows is usually compiled

> (wrt 64 bit integer support). Without the

> > latter it's a fairly

> moot point anyway. > > perl -V says: > > intsize=4, longsize=4, > ptrsize=4, doublesize=8, byteorder=1234 > d_longlong=undef, > longlongsize=8, d_longdbl=define, longdblsize=8 > ivtype='long', > ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8 > alignbytes=8, prototype=define > > (a 64 bit perl says: > intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678 > d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8 > ivtype='__int64', ivsize=8, nvtype='double', nvsize=8, > Off_t='__int64', lseeksize=8 > alignbytes=8, prototype=define > ) > As I understand that, integer scalars are 32 bits, but perl itself > supports 64 bit offsets in files. Which is not a problem, since > floating point scalars are of type double and have a 53 bit > mantissa, which even with rounding should be ok for 40 or even 48 > bit file offsets. > > Also the Windows-64 Version of MapViewOfFile > has exactly the same DWORD arguments for the offset (i.e. 32 bits), > so bits must be shifted anyway to pass a 64 bit offset -- the same > with the length arguments of CreateFileMapping. With > MapViewOfFileEx you can at least specify the length as SIZE_T, > which is 64 bits on Windows-64. > > Ralf

Hi Ralf, I've added large files support for windows (32 and 64 bit) in File::Map 0.42 :-) Leon

Tue Jan 24 17:26:41 2012 LEONT [...] cpan.org - Status changed from 'open' to 'resolved'