Skip Menu |

This queue is for tickets about the ExtUtils-MakeMaker CPAN distribution.

Report information
The Basics
Id: 78395
Status: open
Priority: 0/
Queue: ExtUtils-MakeMaker

People
Owner: ETJ [...] cpan.org
Requestors:
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: (no value)
Fixed in: (no value)



Subject: Win32: Do not use dlltool to create DLLs
Using dlltool instead of gcc/ld cause the following problem in some cases: ----- Error: Can't load '<Module>.dll' for module <Module>: load_file:Invalid access to memory location at <Path>/lib/DynaLoader.pm line xxx ----- This is due to dlltool being depreciated ([1] and [2]), and instead ld/gcc should be used with --enable-auto-image-base. Even with recent binutils (2.22) this is still an issue in rare cases, especially when stripping the DLL. [1] http://sourceforge.net/mailarchive/message.php?msg_id=29151503 [2] http://sourceforge.net/mailarchive/message.php?msg_id=19025935
Subject: 0001-Win32-Do-not-use-dlltool-to-create-DLLs.patch
From ddca82edd6d37255aaa8b7d654df4e38000e3284 Mon Sep 17 00:00:00 2001 From: Martin Schlemmer <Martin.Schlemmer@nwu.ac.za> Date: Mon, 16 Jul 2012 13:06:10 +0200 Subject: [PATCH] Win32: Do not use dlltool to create DLLs Using dlltool instead of gcc/ld cause the following problem in some cases: ----- Error: Can't load '<Module>.dll' for module <Module>: load_file:Invalid access to memory location at <Path>/lib/DynaLoader.pm line xxx ----- This is due to dlltool being depreciated ([1] and [2]), and instead ld/gcc should be used with --enable-auto-image-base. Even with recent binutils (2.22) this is still an issue in rare cases, especially when stripping the DLL. [1] http://sourceforge.net/mailarchive/message.php?msg_id=29151503 [2] http://sourceforge.net/mailarchive/message.php?msg_id=19025935 --- lib/ExtUtils/MM_Win32.pm | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/lib/ExtUtils/MM_Win32.pm b/lib/ExtUtils/MM_Win32.pm index 43ae9a4..42b0cf6 100644 --- a/lib/ExtUtils/MM_Win32.pm +++ b/lib/ExtUtils/MM_Win32.pm @@ -350,10 +350,9 @@ $(INST_DYNAMIC): $(OBJECT) $(MYEXTLIB) $(BOOTSTRAP) $(INST_ARCHAUTODIR)$(DFSEP). '); if ($GCC) { push(@m, - q{ }.$DLLTOOL.q{ --def $(EXPORT_LIST) --output-exp dll.exp - $(LD) -o $@ -Wl,--base-file -Wl,dll.base $(LDDLFLAGS) }.$ldfrom.q{ $(OTHERLDFLAGS) $(MYEXTLIB) $(PERL_ARCHIVE) $(LDLOADLIBS) dll.exp - }.$DLLTOOL.q{ --def $(EXPORT_LIST) --base-file dll.base --output-exp dll.exp - $(LD) -o $@ $(LDDLFLAGS) }.$ldfrom.q{ $(OTHERLDFLAGS) $(MYEXTLIB) $(PERL_ARCHIVE) $(LDLOADLIBS) dll.exp }); + q{ $(LD) $(EXPORT_LIST) -o $@ $(LDDLFLAGS) }.$ldfrom.q{ } + .q{$(OTHERLDFLAGS) $(MYEXTLIB) $(PERL_ARCHIVE) $(LDLOADLIBS) } + .q{-Wl,--enable-auto-image-base }); } elsif ($BORLAND) { push(@m, q{ $(LD) $(LDDLFLAGS) $(OTHERLDFLAGS) }.$ldfrom.q{,$@,,} @@ -390,7 +389,7 @@ gcc. Otherwise, take out all *.pdb files. sub extra_clean_files { my $self = shift; - return $GCC ? (qw(dll.base dll.exp)) : ('*.pdb'); + return $GCC ? (q{}) : ('*.pdb'); } =item init_linker -- 1.7.11.msysgit.1
Subject: Re: [rt.cpan.org #78395] Win32: Do not use dlltool to create DLLs
Date: Mon, 16 Jul 2012 13:50:06 -0700
To: bug-ExtUtils-MakeMaker [...] rt.cpan.org
From: Michael G Schwern <schwern [...] pobox.com>
I'm not very familiar with Windows build tools, but I cannot find anything like an official reference that dlltool is deprecated. Here's the official docs from the latest stable of binutils on dlltool. There's no mention of it being deprecated. http://sourceware.org/binutils/docs-2.22/binutils/dlltool.html#dlltool That said, whether dlltool is deprecated is irrelevant if there's a better way to do it. MakeMaker's code for compiling on Windows can use some love. So, how does one reproduce your problem and any ideas on how to fix it?
From: Martin.Schlemmer [...] nwu.ac.za
On Mon Jul 16 16:50:29 2012, schwern@pobox.com wrote: Show quoted text
> That said, whether dlltool is deprecated is irrelevant if there's a > better way > to do it. MakeMaker's code for compiling on Windows can use some > love. > > So, how does one reproduce your problem and any ideas on how to fix > it?
Currently I can only reproduce it with Cairo-GObject, so not sure if I can produce a test case. Also I mostly use ActiveState Perl, as its easier for our setup that I only maintain the Gtk2 stack, and some other bits not on AS PPM repository, but I can try to get a better test case with Strawberry if needed? Anyhow, the fix is the attached patch - do not jump through all the dlltool/$LD/dlltool/$LD to determine the export list and load address, but just call $LD with the .def and --enable-auto-image-base as arguments.
Subject: Re: [rt.cpan.org #78395] Win32: Do not use dlltool to create DLLs
Date: Tue, 17 Jul 2012 14:42:19 -0700
To: bug-ExtUtils-MakeMaker [...] rt.cpan.org
From: Michael G Schwern <schwern [...] pobox.com>
On 2012.7.17 10:23 AM, Martin Schlemmer via RT wrote: Show quoted text
> Currently I can only reproduce it with Cairo-GObject, so not sure if I > can produce a test case. Also I mostly use ActiveState Perl, as its > easier for our setup that I only maintain the Gtk2 stack, and some other > bits not on AS PPM repository, but I can try to get a better test case > with Strawberry if needed?
AFAIK Strawberry uses a different compilation chain. ActiveState depends on the Microsoft build tools and Strawberry uses MinGW. Both need attention and both are pretty opaque to me. Show quoted text
> Anyhow, the fix is the attached patch - do not jump through all the > dlltool/$LD/dlltool/$LD to determine the export list and load address, > but just call $LD with the .def and --enable-auto-image-base as arguments.
Any potential down sides? Does this require relatively up to date build tools?
From: Martin.Schlemmer [...] nwu.ac.za
On Tue Jul 17 17:42:33 2012, schwern@pobox.com wrote: Show quoted text
> AFAIK Strawberry uses a different compilation chain. ActiveState > depends on the Microsoft build tools >
Right, but it works well with MinGW if you create a LD compatible import libraries for Perl and other modules needed and build with -mms-bitfields in OPTIMIZE. Show quoted text
> and Strawberry uses MinGW. >
I thought it might be a problem with my setup using AS Perl, so I did some testing with Strawberry (5.14.2). Initially I could not reproduce the problem, as Strawberry build with C/LD-FLAGS += " -s ". If I however do: ----- perl Makefile.PL sed -i -e "s: -s : :g" Makefile dmake dmake test strip blib/arch/auto/Cairo/GObject/GObject.dll dmake test ----- The last test fails again with the message in initial post. Applying the fix, above works again, and also the rest of the Gtk2 stack builds and works fine. Show quoted text
> Both need attention and both are pretty opaque to me. >
TBH, for MSFT stuff I cannot really contribute, as I am a Perl/Linux dev that had to switch to Windows in 2007 due to work. However, the bits the patch touch is MinGW (Cygwin?) only, so will not break things for people building with Visual Studio or the Windows SDK. Show quoted text
> > Anyhow, the fix is the attached patch - do not jump through all the > > dlltool/$LD/dlltool/$LD to determine the export list and load
> address,
> > but just call $LD with the .def and --enable-auto-image-base as
> arguments. > > Any potential down sides? Does this require relatively up to date > build tools? >
When I started building Gtk2/Gtk2-Perl in 2007 on Windows, I looked at Tor's stuff, which already used only LD with --enable-auto-image-base (older libtool's did not handle DLLs correctly, so you had to force it). I did a search just now, and there is references of --enable-auto-image-base back to 2002/2003/2005 from the first few pages, and most of those are gcc/libtool/other libraries, and not binutils. According to above, I would say no to potential down sides and up to date build tools. Lastly, MinGW GCC versions have for LINK_SPEC (added back in 2002-12-05): %{shared|mdll: -e _DllMainCRTStartup@12 --enable-auto-image-base} so I think this dlltool usage is really only due to supporting GCC versions before 3.2.3/3.3.1 (if I read the ChangeLog's correctly).
On Mon Jul 16 16:50:29 2012, schwern@pobox.com wrote: Show quoted text
> So, how does one reproduce your problem and any ideas on how to fix > it?
Just another data point: I'm trying to install Wx in a fresh ActivePerl 5.16.3 (build 1604) installation. ActivePerl automatically installs ancient MinGW 3.4.5 compiler and so this is what is used for building non-pure Perl modules. And I am running into another problem with dlltool: the build of Wx fails with this error: c:\Perl\site\lib\auto\MinGW\bin\dlltool.exe --def RichText.def --base-file dll.base --output-exp dll.exp c:\Perl\site\lib\auto\MinGW\bin\dlltool.exe: Unable to open base-file: dll.base dmake.exe: Error code 129, while making '..\..\blib\arch\auto\Wx\RichText\RichText.dll' I have no idea what is supposed to happen here as the file dll.base doesn't exist and doesn't seem to be generated by anything. However, not really surprisingly, but rather helpfully, applying the patch above dropping the use of dlltool completely fixes the problem. Once again, I'm doing this in a completely fresh installation of Windows 7 with just ActivePerl installed, so this should be 100% reproducible if you are still looking for a way to reproduce the problem.
This bug is not a bug in dlltool. Its a bug in the relocations that ld produces for the dll. It only happens on AS's 3.4.5 Vista special r3 compiler. here is a test case written by mithaldu a long time ago showing the crash without using dlltool. Any instance of "load_file:Invalid access to memory location at" from dynaloader means a SEGV occured in DllMain in LoadLibrary, but it was trapped (using SEH) and turned an error code and the DLL was unloaded from memory. 99% of XS modules do not define their own DllMain, sine that is a Win32 only feature, so the DllMain code and a SEGV in it is strictly the responsibility of the C compiler authors.
Subject: Re: [rt.cpan.org #78395] Win32: Do not use dlltool to create DLLs
Date: Mon, 22 Dec 2014 01:49:56 +0200
To: <bug-ExtUtils-MakeMaker [...] rt.cpan.org>
From: "Martin Schlemmer" <Martin.Schlemmer [...] nwu.ac.za>
Martin Schlemmer recently passed away. Please contact one of the following persons for any enquiries. Martin Puttkammer 018 299 1512 martin.puttkammer@nwu.ac.za Roald Eiselen 018 299 1512 roald.eiselen@nwu.ac.za Charlene Schoeman 018 299 1509 charlene.schoeman@nwu.ac.za Show quoted text
>>> "Daniel Dragan via RT" <bug-ExtUtils-MakeMaker@rt.cpan.org> 12/22/14 01:49 >>>
<URL: https://rt.cpan.org/Ticket/Display.html?id=78395 > This bug is not a bug in dlltool. Its a bug in the relocations that ld produces for the dll. It only happens on AS's 3.4.5 Vista special r3 compiler. here is a test case written by mithaldu a long time ago showing the crash without using dlltool. Any instance of "load_file:Invalid access to memory location at" from dynaloader means a SEGV occured in DllMain in LoadLibrary, but it was trapped (using SEH) and turned an error code and the DLL was unloaded from memory. 99% of XS modules do not define their own DllMain, sine that is a Win32 only feature, so the DllMain code and a SEGV in it is strictly the responsibility of the C compiler authors. Vrywaringsklousule / Disclaimer: http://www.nwu.ac.za/it/gov-man/disclaimer.html
On Mon Jul 16 07:07:24 2012, azarah wrote: Show quoted text
> Using dlltool instead of gcc/ld cause the following problem in some cases: > > ----- > Error: Can't load '<Module>.dll' for module <Module>: load_file:Invalid > access to memory location at <Path>/lib/DynaLoader.pm line xxx > ----- > > This is due to dlltool being depreciated ([1] and [2]), and instead > ld/gcc should > be used with --enable-auto-image-base. Even with recent binutils (2.22) > this is > still an issue in rare cases, especially when stripping the DLL. > > [1] http://sourceforge.net/mailarchive/message.php?msg_id=29151503 > [2] http://sourceforge.net/mailarchive/message.php?msg_id=19025935
The OP is gone, but under no circumstances is "--enable-auto-image-base" a replacement for having reloc info on DLLs. Hashes collide. The amount of slots in a 32 bit process is very small, 0+0x0FFC0000/0x00010000 = 4092 slots http://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=ld/emultempl/pe.em;h=60882cef9095c174a17224d0af8355ab51637701;hb=HEAD#l143 http://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob;f=ld/emultempl/pe.em;h=60882cef9095c174a17224d0af8355ab51637701;hb=HEAD#l945 . Each DLL will be mapped starting at a multiple of 64KB, per rules of Win32/64 Memory Manager. Now some DLLs are really fat, like OpenSSL. libeay32.dll is 1172KB mapped + 44KB (padding to next boundary) = 1216KB, ssleay32.dll is 284KB+36KB=320KB. From a real strawberry 5.18 process "63300000" "Image" "1,520" "1,520" "44" "1,168" "28" "1,140" "" "" "10" "Execute/Read/Write" "C:\sperl\perl\bin\perl518.dll" "6347C000" "Unusable" "16" "16" "" "" "" "" "" "" "" "" "" "6CEC0000" "Image" "120" "120" "20" "48" "12" "36" "" "" "9" "Execute/Read" "C:\sperl\perl\bin\libgcc_s_sjlj-1.dll" "6CEDE000" "Unusable" "8" "8" "" "" "" "" "" "" "" "" "" "6FC40000" "Image" "832" "832" "44" "108" "24" "84" "" "" "11" "Execute/Read" "C:\sperl\perl\bin\libstdc++-6.dll" "70840000" "Image" "76" "76" "24" "48" "12" "36" "" "" "9" "Execute/Read" "C:\sperl\perl\lib\auto\Win32\Win32.dll" "70853000" "Unusable" "52" "52" "" "" "" "" "" "" "" "" "" so summing all of that up means 2624 KB are gone automatically in a typical GCC Win32 Perl from the window in which GCC lays down hashed base addresses, so the odds of a hash collision are (0x0FFC0000 - (2624 * 0x10000)) / 0x00010000 = 0x5BC = 1468 64 KB slots 1:1468 for a tiny, <= 64 KB DLL, means a pretty good chance of a collision, and it is nearly guaranteed for a fat XS DLL so there is no way to say "we dont need reloc info on DLLs with GCC Win32 Perl" unless you like playing 1:1468 odds So, here is sample module, that will build on EUMM, but the XS DLL can never be loaded into a process, because of the 2 runs of dlltool, and 2 runs of g++ as LD, create a junk binary when combined with declspec(dllexport). Unlike my last post where I blamed Mingw GCC's 3.4.5's linker, this DLL won't execute with strawberry 5.18/gcc 4.6.3. The specific reason is, that the reloc info appears twice in the final DLL. I've attached a screenshot of the actual SEGV inside DLL main and the fault address. The DLL was originally at, 0x00400000, but there was a conflict, so it was relocated to (picked by OS) 0x00970000. The fault address is 0x00EE5014. Now look at this equation, 0x00970000 - 0x00400000 = 0x570000 new addr - orig addr = adjustment offset by OS orig addr + adjustment offset +adjustment offset (why on earth twice?) 0x00400000 + 0x570000 + 0x570000 = 0xEE0000 0xEE5014 is the SEGV address So it means there were 2 relocation entries for after relocation machine code address 0x00971429, notice 0x00971427 is not the place to do the relocation, 0xC7 0x05 is the x86 opcode, the next 4 bytes "14 50 EE 00" are the destination operand, so 0x00971429 is the correct place for the relocation, not 0x00971427. Now to verify the hypothesis with "objdump -p blib\arch/auto/GCC/BADDLL/BADDLL.dll" which i've attached a full dump of --------------------------------------- ****************cut******************** PE File Base Relocations (interpreted .reloc section contents) Virtual Address: 00001000 Chunk size 324 (0x144) Number of fixups 158 reloc 0 offset 1d [101d] HIGHLOW ****************cut******************** reloc 29 offset 2c1 [12c1] HIGHLOW reloc 30 offset 429 [1429] HIGHLOW reloc 31 offset 467 [1467] HIGHLOW ****************cut******************** Virtual Address: 00001000 Chunk size 320 (0x140) Number of fixups 156 reloc 0 offset 1d [101d] HIGHLOW ****************cut******************** reloc 29 offset 2c1 [12c1] HIGHLOW reloc 30 offset 429 [1429] HIGHLOW reloc 31 offset 467 [1467] HIGHLOW ****************cut******************** --------------------------------------- yep, 2 relocation entries for offset into the DLL 0x1429
Subject: GCC-BADDLL-0.01.zip
Download GCC-BADDLL-0.01.zip
application/x-zip-compressed 5.8k

Message body not shown because it is not plain text.

Subject: gcc bad reloc.PNG
Download gcc bad reloc.PNG
image/png 62.6k
gcc bad reloc.PNG
Subject: objdump -p dump.txt

Message body is not shown because it is too large.

Now to explain the Mingw GCC EUMM build process a little bit (as much as I know, and I dont know very much except from trial and error today), this is the latter half of "dmake -n test", with Straberry 5.18's gcc 4.6.3 C:\sperl\perl\bin\perl.exe C:\sperl\perl\lib\ExtUtils\xsubpp -typemap C:\sperl\perl\lib\ExtUtils\typemap BADDLL.xs > BADDLL.xsc && C:\sperl\perl\bin\perl.exe -MExtUtils::Command -e mv -- BADDLL.xsc BADDLL.c make the .xs into a .c, ok gcc -c -I. -s -O2 -DWIN32 -DPERL_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fno-strict-aliasing -mms-bitfields -s -O2 -DVERSION=\"0.01\" -DXS_VERSION=\"0.01\" "-IC:\sperl\perl\lib\CORE" BADDLL.c turn .c into .o, ok C:\sperl\perl\bin\perl.exe -l -e "print qq{@ARGV}" -- "Running Mkbootstrap for GCC::BADDLL ()" C:\sperl\perl\bin\perl.exe "-MExtUtils::Mkbootstrap" -e "Mkbootstrap('BADDLL','');" C:\sperl\perl\bin\perl.exe -MExtUtils::Command -e touch -- BADDLL.bs C:\sperl\perl\bin\perl.exe -MExtUtils::Command -e chmod -- 644 BADDLL.bs make .bs file, ok C:\sperl\perl\bin\perl.exe -MExtUtils::Mksymlists -e "Mksymlists('NAME'=>\"GCC::BADDLL\", 'DLBASE' => 'BADDLL', 'DL_FUNCS' => { }, 'FUNCLIST' => [q[boot_GCC__BADDLL]], 'IMPORTS' => { }, 'DL_VARS' => []);" make .def file dlltool --def BADDLL.def --output-exp dll.exp take in ASCII .def, make binary COFF dll.exp file g++ -o blib\arch\auto\GCC\BADDLL\BADDLL.dll -Wl,--base-file -Wl,dll.base -mdll -s -L"C:\sperl\perl\lib\CORE" -L"C:\sperl\c\lib" BADDLL.o -Wl,--image-base=0x00400000 C:\sperl\perl\lib\CORE\libperl518.a -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 dll.exp this is where the build process differs, based on whether function "getfive" had declspec(dllexport) or not, 2 things are made here -dll.base fil, the dll.base is binary, and is not a COFF file - DLL blib\arch\auto\GCC\BADDLL\BADDLL.dll if the baddll.c/baddll.xs DOES NOT have declspec(dllexport), then dll blib\arch\auto\GCC\BADDLL\BADDLL.dll DOES NOT have a relocation table, it will only be loadedable into a process if no base address conflict if the baddll.c/baddll.xs DOES have declspec(dllexport), then dll blib\arch\auto\GCC\BADDLL\BADDLL.dll DOES have a relocation table, with NO duplicate reloc entries, ready to run dlltool --def BADDLL.def --base-file dll.base --output-exp dll.exp take in baddll.def and dll.base, output dll.exp g++ -o blib\arch\auto\GCC\BADDLL\BADDLL.dll -mdll -s -L"C:\sperl\perl\lib\CORE" -L"C:\sperl\c\lib" BADDLL.o -Wl,--image-base=0x00400000 C:\sperl\perl\lib\CORE\libperl518.a -lmoldname -lkernel32 -luser32 -lgdi32 -lwinspool -lcomdlg32 -ladvapi32 -lshell32 -lole32 -loleaut32 -lnetapi32 -luuid -lws2_32 -lmpr -lwinmm -lversion -lodbc32 -lodbccp32 -lcomctl32 dll.exp create blib\arch\auto\GCC\BADDLL\BADDLL.dll again, this time not producing dll.base, and taking in a new, 2nd version of dll.exp this is again where the build process differs if the baddll.c/baddll.xs DOES NOT have declspec(dllexport), then dll blib\arch\auto\GCC\BADDLL\BADDLL.dll gets a relocation table, now its ready to run if the baddll.c/baddll.xs DOES have declspec(dllexport), then dll blib\arch\auto\GCC\BADDLL\BADDLL.dll gets 2 copies of a relocation table, with duplicate reloc entries, and this is a junk dll/binary, this dll IS NOT ready to run and is permanently broken C:\sperl\perl\bin\perl.exe -MExtUtils::Command -e chmod -- 755 blib\arch\auto\GCC\BADDLL\BADDLL.dll C:\sperl\perl\bin\perl.exe -MExtUtils::Command -e rm_rf -- blib\arch\auto\GCC\BADDLL\BADDLL.bs C:\sperl\perl\bin\perl.exe -MExtUtils::Command -e cp -- BADDLL.bs blib\arch\auto\GCC\BADDLL\BADDLL.bs C:\sperl\perl\bin\perl.exe -MExtUtils::Command -e chmod -- 644 blib\arch\auto\GCC\BADDLL\BADDLL.bs rem rem rem C:\sperl\perl\bin\perl.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib\lib', 'blib\arch')" t/*.t rem boilerplate, uninteresting