Bug #118053 for PAR-Packer: Problem with packed perl archive for biber on 64-bit Cygwin

Fri Sep 23 15:51:35 2016 kbrown [...] cornell.edu - Ticket created

CC:	Kime Philip <philkime [...] kime.org.uk>, David Carlisle <d.p.carlisle [...] gmail.com>
Subject:	Problem with packed perl archive for biber on 64-bit Cygwin
Date:	Fri, 23 Sep 2016 15:51:18 -0400
To:	bug-PAR-Packer [...] rt.cpan.org
From:	Ken Brown <kbrown [...] cornell.edu>

Hi Roderich, Phil Kime suggested that I raise a ticket concerning a problem I'm having with a packed archive for biber (https://sourceforge.net/projects/biblatex-biber/) on 64-bit Cygwin. The problem doesn't occur on 32-bit Cygwin. Here's the short version: I use PAR::Packer to create a packed archive biber.exe. It unpacks itself and runs fine in the Cygwin distribution on which I created it. But if I run it on a fresh, bare-bones Cygwin installation (which in particular does not have perl installed), it fails as follows: $ ./biber.exe C:/cygwin64a/tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/biber: error while loading shared libraries: cygperl5_24.dll: cannot open shared object file: No such file or directory When I look at the cache, I see the following: $ ls -l /tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/ total 2136 -rwxr-x--- 1 kbrown None 53248 Sep 23 15:01 biber -rwxr-x--- 1 kbrown None 2133011 Sep 23 15:01 cygperl5_24.dll So nothing has been unpacked except biber and one DLL. And then the system tries to load that DLL and fails, because its dependent DLLs have not yet been unpacked. In this case the DLL cygcrypt-0.dll is the problematic one, as we can see by running cygcheck: $ cygcheck /tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/cygperl5_24.dll C:\cygwin64a\tmp\par-6b62726f776e\cache-128a18f7440676b8f4632879a6f42b099bad8a05\cygperl5_24.dll C:\cygwin64a\bin\cygwin1.dll C:\WINDOWS\system32\KERNEL32.dll C:\WINDOWS\system32\ntdll.dll C:\WINDOWS\system32\KERNELBASE.dll C:\cygwin64a\bin\cygssp-0.dll cygcheck: track_down: could not find cygcrypt-0.dll [Note that cygperl5_24.dll also depends on cygssp-0.dll, which also should have been unpacked; but in this case it doesn't matter, because /usr/bin/cygssp-0.dll exists in the bare-bones Cygwin install.] Do you have any idea why biber.exe doesn't fully unpack itself before trying to load cygperl5_24.dll? The rest of this message provides more details about how I built biber.exe. Please let me know if there's anything else you need. My system (both build and test): $ uname -a CYGWIN_NT-10.0 moufang 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin My perl on the build system: $ perl -v This is perl 5, version 24, subversion 0 (v5.24.0) built for x86_64-cygwin-threads-multi I built this perl myself, because biber requires perl 5.24, and Cygwin's perl is currently at 5.22. I created the build installation by doing the following: 1. Start with a fresh Cygwin installation, install my build of perl-5.24, and install a few other packages that are needed for the next steps. 2. Use cpan to install the build prerequisites for biber and to install PAR::Packer. (I repeated this process today to make sure my build environment was clean, so all modules are at their latest versions as of today.) 3. Clone the biber git repo, checkout v2.6, and build and install biber. 4. Create the packed biber archive by running the attached script build.sh, which uses the attached auxiliary file biber.files. Thanks in advance for any help you can offer. Ken

Message body is not shown because sender requested not to inline it.

Sat Sep 24 21:06:22 2016 kbrown [...] cornell.edu - Correspondence added

CC:	Kime Philip <philkime [...] kime.org.uk>, David Carlisle <d.p.carlisle [...] gmail.com>
Subject:	Re: [rt.cpan.org #118053] AutoReply: Problem with packed perl archive for biber on 64-bit Cygwin
Date:	Sat, 24 Sep 2016 21:06:07 -0400
To:	bug-PAR-Packer [...] rt.cpan.org
From:	Ken Brown <kbrown [...] cornell.edu>

On 9/23/2016 3:51 PM, Bugs in PAR-Packer via RT wrote: Show quoted text

> > Greetings, > > This message has been automatically generated in response to the > creation of a trouble ticket regarding: > "Problem with packed perl archive for biber on 64-bit Cygwin", > a summary of which appears below. > > There is no need to reply to this message right now. Your ticket has been > assigned an ID of [rt.cpan.org #118053]. Your ticket is accessible > on the web at: > > https://rt.cpan.org/Ticket/Display.html?id=118053 > > Please include the string: > > [rt.cpan.org #118053] > > in the subject line of all future correspondence about this issue. To do so, > you may reply to this message. > > Thank you, > bug-PAR-Packer@rt.cpan.org > > ------------------------------------------------------------------------- > Hi Roderich, > > Phil Kime suggested that I raise a ticket concerning a problem I'm > having with a packed archive for biber > (https://sourceforge.net/projects/biblatex-biber/) on 64-bit Cygwin. > The problem doesn't occur on 32-bit Cygwin. > > Here's the short version: I use PAR::Packer to create a packed archive > biber.exe. It unpacks itself and runs fine in the Cygwin distribution > on which I created it. But if I run it on a fresh, bare-bones Cygwin > installation (which in particular does not have perl installed), it > fails as follows: > > $ ./biber.exe > C:/cygwin64a/tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/biber: > error while loading shared libraries: cygperl5_24.dll: cannot open > shared object file: No such file or directory > > When I look at the cache, I see the following: > > $ ls -l > /tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/ > total 2136 > -rwxr-x--- 1 kbrown None 53248 Sep 23 15:01 biber > -rwxr-x--- 1 kbrown None 2133011 Sep 23 15:01 cygperl5_24.dll > > So nothing has been unpacked except biber and one DLL. And then the > system tries to load that DLL and fails, because its dependent DLLs have > not yet been unpacked. In this case the DLL cygcrypt-0.dll is the > problematic one, as we can see by running cygcheck: > > $ cygcheck > /tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/cygperl5_24.dll > C:\cygwin64a\tmp\par-6b62726f776e\cache-128a18f7440676b8f4632879a6f42b099bad8a05\cygperl5_24.dll > C:\cygwin64a\bin\cygwin1.dll > C:\WINDOWS\system32\KERNEL32.dll > C:\WINDOWS\system32\ntdll.dll > C:\WINDOWS\system32\KERNELBASE.dll > C:\cygwin64a\bin\cygssp-0.dll > cygcheck: track_down: could not find cygcrypt-0.dll > > [Note that cygperl5_24.dll also depends on cygssp-0.dll, which also > should have been unpacked; but in this case it doesn't matter, because > /usr/bin/cygssp-0.dll exists in the bare-bones Cygwin install.] > > Do you have any idea why biber.exe doesn't fully unpack itself before > trying to load cygperl5_24.dll? > > The rest of this message provides more details about how I built > biber.exe. Please let me know if there's anything else you need. > > My system (both build and test): > > $ uname -a > CYGWIN_NT-10.0 moufang 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin > > My perl on the build system: > > $ perl -v > > This is perl 5, version 24, subversion 0 (v5.24.0) built for > x86_64-cygwin-threads-multi > > I built this perl myself, because biber requires perl 5.24, and Cygwin's > perl is currently at 5.22. > > I created the build installation by doing the following: > > 1. Start with a fresh Cygwin installation, install my build of > perl-5.24, and install a few other packages that are needed for the next > steps. > > 2. Use cpan to install the build prerequisites for biber and to install > PAR::Packer. (I repeated this process today to make sure my build > environment was clean, so all modules are at their latest versions as of > today.) > > 3. Clone the biber git repo, checkout v2.6, and build and install biber. > > 4. Create the packed biber archive by running the attached script > build.sh, which uses the attached auxiliary file biber.files. > > Thanks in advance for any help you can offer. > > Ken

I understand a little better how the packed executable is supposed to work. First the embedded files are extracted, which are a perl interpreter (named biber) and cygperl5_24.dll in; it's then supposed to be possible to run the extracted biber, which will extract the rest of the files. Roderich, is this right? So it seems that the solution should be to add cygcrypt-0.dll as an embedded file. I haven't tried to do this in the PAR::Packer source yet but I did test the theory as follows: I installed the libcrypt0 package (which provides cygcrypt-0.dll) in my bare-bones test installation, and biber.exe then unpacked itself and ran. The reason I didn't see this problem in 32-bit Cygwin is that for some reason my 32-bit build of cygperl5_24.dll doesn't depend on cygcrypt-0.dll. Roderich, should the build of PAR::Packer have detected the need to embed cygcrypt-0.dll? If not, what's the right fix for this? David, I don't think this explains your symptoms. I'll write separately to you about that, since I don't think it's related to this PAR::Packer issue. Ken

Sun Sep 25 03:06:46 2016 RSCHUPP [...] cpan.org - Correspondence added

On 2016-09-24 21:06:22, kbrown@cornell.edu wrote: Show quoted text

> On 9/23/2016 3:51 PM, Bugs in PAR-Packer via RT wrote: > I understand a little better how the packed executable is supposed to > work. First the embedded files are extracted, which are a perl > interpreter (named biber) and cygperl5_24.dll in; it's then supposed > to > be possible to run the extracted biber, which will extract the rest of > the files. Roderich, is this right?

Your analysis is correct. Show quoted text

> So it seems that the solution should be to add cygcrypt-0.dll as an > embedded file.

Yes. Try changing line 43 of myldr/embed_files.pl in the PAR::Packer source to *is_system_lib = sub { shift =~ m{^/usr/bin/(?!cygcrypt\b)|^\Q$system_root\E/}i }; i.e. /usr/bin/cygcrypt-0.dll is _not_ regarded as a "system library" anymore. Then re-build and install PAR:Packer and re-pack biber. Cheers, Roderich

Sun Sep 25 03:06:48 2016 The RT System itself - Status changed from 'new' to 'open'

Sun Sep 25 11:24:43 2016 kbrown [...] cornell.edu - Correspondence added

Subject:	Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Date:	Sun, 25 Sep 2016 11:24:27 -0400
To:	bug-PAR-Packer [...] rt.cpan.org
From:	Ken Brown <kbrown [...] cornell.edu>

On 9/25/2016 3:06 AM, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 > > > On 2016-09-24 21:06:22, kbrown@cornell.edu wrote:

>> On 9/23/2016 3:51 PM, Bugs in PAR-Packer via RT wrote: >> I understand a little better how the packed executable is supposed to >> work. First the embedded files are extracted, which are a perl >> interpreter (named biber) and cygperl5_24.dll in; it's then supposed >> to >> be possible to run the extracted biber, which will extract the rest of >> the files. Roderich, is this right?

> > Your analysis is correct. >

>> So it seems that the solution should be to add cygcrypt-0.dll as an >> embedded file.

> > Yes. Try changing line 43 of myldr/embed_files.pl in the PAR::Packer source to > > *is_system_lib = sub { shift =~ m{^/usr/bin/(?!cygcrypt\b)|^\Q$system_root\E/}i }; > > i.e. /usr/bin/cygcrypt-0.dll is _not_ regarded as a "system library" anymore. > Then re-build and install PAR:Packer and re-pack biber. > > Cheers, Roderich

Thanks, that fixes it. Ken

Sun Sep 25 13:37:42 2016 kbrown [...] cornell.edu - Correspondence added

CC:	Philip Kime <philkime [...] kime.org.uk>, David Carlisle <d.p.carlisle [...] gmail.com>
Subject:	Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Date:	Sun, 25 Sep 2016 13:37:28 -0400
To:	bug-PAR-Packer [...] rt.cpan.org
From:	Ken Brown <kbrown [...] cornell.edu>

On 9/25/2016 11:24 AM, Ken Brown wrote: Show quoted text

> On 9/25/2016 3:06 AM, Roderich Schupp via RT wrote:

>> <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 > >> >> On 2016-09-24 21:06:22, kbrown@cornell.edu wrote:

>>> On 9/23/2016 3:51 PM, Bugs in PAR-Packer via RT wrote: >>> I understand a little better how the packed executable is supposed to >>> work. First the embedded files are extracted, which are a perl >>> interpreter (named biber) and cygperl5_24.dll in; it's then supposed >>> to >>> be possible to run the extracted biber, which will extract the rest of >>> the files. Roderich, is this right?

>> >> Your analysis is correct. >>

>>> So it seems that the solution should be to add cygcrypt-0.dll as an >>> embedded file.

>> >> Yes. Try changing line 43 of myldr/embed_files.pl in the PAR::Packer >> source to >> >> *is_system_lib = sub { shift =~ >> m{^/usr/bin/(?!cygcrypt\b)|^\Q$system_root\E/}i }; >> >> i.e. /usr/bin/cygcrypt-0.dll is _not_ regarded as a "system library" >> anymore. >> Then re-build and install PAR:Packer and re-pack biber. >> >> Cheers, Roderich

> > Thanks, that fixes it.

Actually, there's still one glitch. If I run biber with no arguments, it unpacks itself and gives me a usage message. It also works fine if I run 'biber --help'. But if I give it a file as an argument (e.g., 'biber test.bcf') before the cache is created, it unpacks itself and then hangs. After I kill it, I can rerun it with no problem. David Carlisle, who is also testing, finds that it still hangs even after the cache is created, but that may be a separate issue. Do you have any idea what might cause this? For me it's just a glitch. For David, it means that biber is unusable. Thanks. Ken

Mon Sep 26 06:16:57 2016 RSCHUPP [...] cpan.org - Correspondence added

Am 2016-09-25 13:37:42, kbrown@cornell.edu schrieb: Show quoted text

> Actually, there's still one glitch. If I run biber with no arguments, > it unpacks itself and gives me a usage message. It also works fine if I > run 'biber --help'. But if I give it a file as an argument (e.g., > 'biber test.bcf') before the cache is created, it unpacks itself and > then hangs. After I kill it, I can rerun it with no problem.

Sorry, no clue. If "biber --help" works, then basically PAR::Packer's job is done (though there might still be missing files etc, but typically that doesn't result in a hang). Does this hang occur also on the machine where you packed biber? Cheers, Roderich

Mon Sep 26 09:51:06 2016 kbrown [...] cornell.edu - Correspondence added

Subject:	Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Date:	Mon, 26 Sep 2016 09:50:49 -0400
To:	bug-PAR-Packer [...] rt.cpan.org
From:	Ken Brown <kbrown [...] cornell.edu>

On 9/26/2016 6:17 AM, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 > > > Am 2016-09-25 13:37:42, kbrown@cornell.edu schrieb:

>> Actually, there's still one glitch. If I run biber with no arguments, >> it unpacks itself and gives me a usage message. It also works fine if I >> run 'biber --help'. But if I give it a file as an argument (e.g., >> 'biber test.bcf') before the cache is created, it unpacks itself and >> then hangs. After I kill it, I can rerun it with no problem.

> > Sorry, no clue. If "biber --help" works, then basically PAR::Packer's job > is done (though there might still be missing files etc, but typically that > doesn't result in a hang). Does this hang occur also on the machine > where you packed biber?

It did originally, but I just repacked biber (which a slight change), and now I can't reproduce the problem anymore. The change was to remove these two arguments from the invocation of pp: --link=/usr/bin/cygssp-0.dll --link=/usr/bin/cygcrypt-0.dll The first is not needed because cygssp-0.dll is contained in a minimal Cygwin install, and the second is not needed now that cygcrypt-0.dll is embedded. I'm not sure whether (or why) these changes made a difference, but in any case the problem seems to be gone. Thanks for your help. Ken

Mon Sep 26 13:16:13 2016 roderich.schupp [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Date:	Mon, 26 Sep 2016 19:15:57 +0200
To:	bug-PAR-Packer [...] rt.cpan.org
From:	Roderich Schupp <roderich.schupp [...] gmail.com>

On Mon, Sep 26, 2016 at 3:51 PM, Ken Brown via RT < bug-PAR-Packer@rt.cpan.org> wrote: Show quoted text

> The change was to remove > these two arguments from the invocation of pp: > > --link=/usr/bin/cygssp-0.dll > --link=/usr/bin/cygcrypt-0.dll >

I was going to suggest that as well: these result in a second copy of the DLL to be unpacked - while the first copy is probably already in use. Which Windows doesn't allow i(the Linux equivalent error would be EBUSY). But then I convinced myself that the two copies would go to different directories (the first to the root of the cache area, the second to cache/shlibs or similar)... Cheers, Roderich

Mon Sep 26 13:23:05 2016 kbrown [...] cornell.edu - Correspondence added

Subject:	Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Date:	Mon, 26 Sep 2016 13:23:00 -0400
To:	bug-PAR-Packer [...] rt.cpan.org
From:	Ken Brown <kbrown [...] cornell.edu>

On 9/26/2016 1:16 PM, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 > > > On Mon, Sep 26, 2016 at 3:51 PM, Ken Brown via RT < > bug-PAR-Packer@rt.cpan.org> wrote: >

>> The change was to remove >> these two arguments from the invocation of pp: >> >> --link=/usr/bin/cygssp-0.dll >> --link=/usr/bin/cygcrypt-0.dll >>

> > I was going to suggest that as well: these result in a second copy of the > DLL to be unpacked - while the first copy is probably already in use. > Which Windows doesn't allow i(the Linux equivalent error would be EBUSY). > But then I convinced myself that the two copies would go to different > directories (the first to the root of the cache area, the second to > cache/shlibs or similar)...

No, they were actually both going to the root of the cache directory. So you're explanation makes perfect sense. Thanks. Ken

Mon Sep 26 13:34:36 2016 roderich.schupp [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Date:	Mon, 26 Sep 2016 19:34:22 +0200
To:	bug-PAR-Packer [...] rt.cpan.org
From:	Roderich Schupp <roderich.schupp [...] gmail.com>

On Mon, Sep 26, 2016 at 7:23 PM, Ken Brown via RT < bug-PAR-Packer@rt.cpan.org> wrote: Show quoted text

> No, they were actually both going to the root of the cache directory. > So you're explanation makes perfect sense. >

You're absolutely right. --link stuff goes into folder "shlib" in the packed executable viewed as a zip, but gets extracted into the root of the cache directory. Cheers, Roderich

Tue Sep 27 11:23:32 2016 kbrown [...] cornell.edu - Correspondence added

Subject:	Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Date:	Tue, 27 Sep 2016 11:23:25 -0400
To:	bug-PAR-Packer [...] rt.cpan.org
From:	Ken Brown <kbrown [...] cornell.edu>

It looks like David Carlisle's issue is completely unrelated, so I think you can apply your change and close this ticket. Thanks again for your help. Ken

Thu Sep 29 15:37:24 2016 RSCHUPP [...] cpan.org - Status changed from 'open' to 'resolved'

Thu Sep 29 15:37:25 2016 RSCHUPP [...] cpan.org - Broken in 1.035 added