Hi Leon-
Here are the lines from Map.xs where the problem comes in.
It appears that the test is against ULONG_MAX which is something
like 2**32 where it should be against ULONGLONG_MAX or
whatever is appropriate.
569 struct mmap_info* magical;
570 if (length > ULONG_MAX - correction)
571 real_croak_pv(aTHX_ "Can't map:
length + offset overflows");
For reference, the File::Map test is run on win7 x64 and 64bit
Strawberry Perl portable.
F:\chm\perl\strawberry\v51631>perl -V
Summary of my perl5 (revision 5 version 16 subversion 3) configuration:
Platform:
osname=MSWin32, osvers=4.0, archname=MSWin32-x64-multi-thread
uname='Win32 strawberry-perl 5.16.3.1 #1 Tue Mar 12 12:12:07 2013 x64'
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=define, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='gcc', ccflags =' -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE
-DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS
-fno-strict-aliasing -mms-bitfields',
optimize='-s -O2',
cppflags='-DWIN32'
ccversion='', gccversion='4.6.3', gccosandvers=''
intsize=4, longsize=4, ptrsize=8, doublesize=8, byteorder=12345678
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long long', ivsize=8, nvtype='double', nvsize=8,
Off_t='long long', lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries:
ld='g++.exe', ldflags ='-s
-L"F:\chm\perl\strawberry\v51631\perl\lib\CORE"
-L"F:\chm\perl\strawberry\v51631\c\lib"'
libpth=F:\chm\perl\strawberry\v51631\c\lib
F:\chm\perl\strawberry\v51631\c\x86_64-w64-mingw32\lib
libs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32
-ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32
-lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
perllibs=-lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool
-lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid
-lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32
libc=, so=dll, useshrplib=true, libperl=libperl516.a
gnulibc_version=''
Dynamic Linking:
dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
cccdlflags=' ', lddlflags='-mdll -s
-L"F:\chm\perl\strawberry\v51631\perl\lib\CORE"
-L"F:\chm\perl\strawberry\v51631\c\lib"'
Characteristics of this binary (from libperl):
Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY
PERLIO_LAYERS PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
PERL_MALLOC_WRAP PERL_PRESERVE_IVUV PL_OP_SLAB_ALLOC
USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES
USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE
USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF
Built under MSWin32
Compiled at Mar 12 2013 12:16:40
%ENV:
PERLDOC_PAGER="less"
@INC:
F:/chm/perl/strawberry/v51631/perl/site/lib
F:/chm/perl/strawberry/v51631/perl/vendor/lib
F:/chm/perl/strawberry/v51631/perl/lib
.
On Tue, May 21, 2013 at 5:47 PM, Leon Timmermans via RT
<bug-File-Map@rt.cpan.org> wrote:
Show quoted text> <URL:
https://rt.cpan.org/Ticket/Display.html?id=85526 >
>
> On Tue, May 21, 2013 at 11:05 PM, Chris Marshall via RT
> <bug-File-Map@rt.cpan.org> wrote:
>> AFAICT, the underlying routines on 64bit OSes do support >32bit sizes and
>> offsets.
>> This limitation prevents PDL from mapping files larger than 4GiB when for these
>> extremely large files, this is exactly where we expect the greatest benefit
>> from mapping.
>>
>> E.g., by using map_anonymous() we can avoid the perl exit on Out-of-memory
>> problem
>> for extremely large memory requests.
>>
>> pdl> use File::Map qw(:map)
>>
>> pdl> map_anonymous $huge, 2_000_000_000, 'private'
>>
>> pdl> p length $huge
>> 2000000000
>>
>> pdl> unmap $huge
>>
>> pdl> map_anonymous $huge, 4_000_000_000, 'private'
>>
>> pdl> p length $huge
>> 4000000000
>>
>> pdl> unmap $huge
>>
>> pdl> map_anonymous $huge, 5_000_000_000, 'private'
>> Runtime error: Can't map: length + offset overflows at (eval 464) line 5.
>
> I took special care to make File::Map 64-bit safe (because of the same
> reasons as you provide), and I can't reproduce this issue on a 64-bit
> OS.
>
> On a 32 bit OS however I would expect this limitation; one can't map a
> file larger than one's memory space. That's a limitation of the
> architecture, not of my software.
>
> Leon
>
