Bug #50909 for ExtUtils-MakeMaker: Overriding INSTALLPRIVLIB causes recursive variable error when PREFIX is set too

Wed Oct 28 06:04:31 2009 shay [...] cpan.org - Ticket created

Subject:

Overriding INSTALLPRIVLIB causes recursive variable error when PREFIX is set too

The attached Makefile.PL creates a valid Makefile when invoked without additional arguments, but when run with a "PREFIX=..." argument it creates a Makefile with a recursive variable definition, namely, INSTALLARCHLIB=$(INSTALLARCHLIB). This problem has come to light as a result of a change made in IO-Compress-2.022 (resulting from discussions on p5p regarding how best to cope with a re-arrangement of IO-Compress's installed file locations), and was reported in CPAN RT #50531. On my system (Win32, VC9, Perl-5.10.1 installed in C:\perl5, EU-MM-6.55_02) the differences between the output of "perl Makefile.PL" and "perl Makefile.PL PREFIX=C:/Temp" are as follows, which is pretty much what I'd expect, except that the PREFIX version has that recursive variable definition. I think it should have INSTALLARCHLIB=$(PERLPREFIX)\lib instead. --- Makefile.normal 2009-10-28 09:42:13.438253200 +0000 +++ Makefile.prefix 2009-10-28 09:42:59.016378200 +0000 @@ -6,7 +6,7 @@ # # ANY CHANGES MADE HERE WILL BE LOST! # -# MakeMaker ARGV: () +# MakeMaker ARGV: (q[PREFIX=C:/Temp]) # # MakeMaker Parameters: @@ -43,8 +43,8 @@ SITELIBEXP = C:\perl5\site\lib SITEARCHEXP = C:\perl5\site\lib SO = dll -VENDORARCHEXP = -VENDORLIBEXP = +VENDORARCHEXP = $(VENDORPREFIX)\lib\5.10.1\MSWin32-x86-multi-thread +VENDORLIBEXP = $(VENDORPREFIX)\lib # --- MakeMaker constants section: @@ -70,45 +70,45 @@ MAN3EXT = 3 INSTALLDIRS = site DESTDIR = -PREFIX = $(SITEPREFIX) -PERLPREFIX = C:\perl5 -SITEPREFIX = C:\perl5\site -VENDORPREFIX = +PREFIX = C:/Temp +PERLPREFIX = $(PREFIX) +SITEPREFIX = $(PREFIX) +VENDORPREFIX = $(PREFIX) INSTALLPRIVLIB = $(INSTALLARCHLIB) DESTINSTALLPRIVLIB = $(DESTDIR)$(INSTALLPRIVLIB) -INSTALLSITELIB = C:\perl5\site\lib +INSTALLSITELIB = $(SITEPREFIX)\lib DESTINSTALLSITELIB = $(DESTDIR)$(INSTALLSITELIB) -INSTALLVENDORLIB = +INSTALLVENDORLIB = $(VENDORPREFIX)\lib DESTINSTALLVENDORLIB = $(DESTDIR)$(INSTALLVENDORLIB) -INSTALLARCHLIB = C:\perl5\lib +INSTALLARCHLIB = $(INSTALLARCHLIB) DESTINSTALLARCHLIB = $(DESTDIR)$(INSTALLARCHLIB) -INSTALLSITEARCH = C:\perl5\site\lib +INSTALLSITEARCH = $(SITEPREFIX)\lib DESTINSTALLSITEARCH = $(DESTDIR)$(INSTALLSITEARCH) -INSTALLVENDORARCH = +INSTALLVENDORARCH = $(VENDORPREFIX)\lib\5.10.1\MSWin32-x86-multi-thread DESTINSTALLVENDORARCH = $(DESTDIR)$(INSTALLVENDORARCH) -INSTALLBIN = C:\perl5\bin +INSTALLBIN = $(PERLPREFIX)\bin DESTINSTALLBIN = $(DESTDIR)$(INSTALLBIN) -INSTALLSITEBIN = C:\perl5\bin +INSTALLSITEBIN = $(SITEPREFIX)\bin DESTINSTALLSITEBIN = $(DESTDIR)$(INSTALLSITEBIN) -INSTALLVENDORBIN = +INSTALLVENDORBIN = $(VENDORPREFIX)\bin DESTINSTALLVENDORBIN = $(DESTDIR)$(INSTALLVENDORBIN) -INSTALLSCRIPT = C:\perl5\bin +INSTALLSCRIPT = $(PERLPREFIX)\bin DESTINSTALLSCRIPT = $(DESTDIR)$(INSTALLSCRIPT) INSTALLSITESCRIPT = $(INSTALLSCRIPT) DESTINSTALLSITESCRIPT = $(DESTDIR)$(INSTALLSITESCRIPT) -INSTALLVENDORSCRIPT = +INSTALLVENDORSCRIPT = $(VENDORPREFIX)\bin DESTINSTALLVENDORSCRIPT = $(DESTDIR)$(INSTALLVENDORSCRIPT) -INSTALLMAN1DIR = C:\perl5\man\man1 +INSTALLMAN1DIR = $(PERLPREFIX)\man\man1 DESTINSTALLMAN1DIR = $(DESTDIR)$(INSTALLMAN1DIR) INSTALLSITEMAN1DIR = $(INSTALLMAN1DIR) DESTINSTALLSITEMAN1DIR = $(DESTDIR)$(INSTALLSITEMAN1DIR) -INSTALLVENDORMAN1DIR = +INSTALLVENDORMAN1DIR = $(VENDORPREFIX)\man\man1 DESTINSTALLVENDORMAN1DIR = $(DESTDIR)$(INSTALLVENDORMAN1DIR) -INSTALLMAN3DIR = C:\perl5\man\man3 +INSTALLMAN3DIR = $(PERLPREFIX)\man\man3 DESTINSTALLMAN3DIR = $(DESTDIR)$(INSTALLMAN3DIR) INSTALLSITEMAN3DIR = $(INSTALLMAN3DIR) DESTINSTALLSITEMAN3DIR = $(DESTDIR)$(INSTALLSITEMAN3DIR) -INSTALLVENDORMAN3DIR = +INSTALLVENDORMAN3DIR = $(VENDORPREFIX)\man\man3 DESTINSTALLVENDORMAN3DIR = $(DESTDIR)$(INSTALLVENDORMAN3DIR) PERL_LIB = C:\perl5\lib PERL_ARCHLIB = C:\perl5\lib @@ -562,7 +562,7 @@ # --- MakeMaker dist_test section: disttest : distdir cd $(DISTVNAME) - $(ABSPERLRUN) Makefile.PL + $(ABSPERLRUN) Makefile.PL "PREFIX=C:/Temp" $(MAKE) $(PASTHRU) $(MAKE) test $(PASTHRU) cd .. @@ -731,7 +731,7 @@ -$(NOECHO) $(RM_F) $(MAKEFILE_OLD) -$(NOECHO) $(MV) $(FIRST_MAKEFILE) $(MAKEFILE_OLD) - $(MAKE) $(USEMAKEFILE) $(MAKEFILE_OLD) clean $(DEV_NULL) - $(PERLRUN) Makefile.PL + $(PERLRUN) Makefile.PL "PREFIX=C:/Temp" $(NOECHO) $(ECHO) "==> Your Makefile has been rebuilt. <==" $(NOECHO) $(ECHO) "==> Please rerun the $(MAKE) command. <==" $(FALSE) @@ -752,7 +752,8 @@ $(NOECHO) $(PERLRUNINST) \ Makefile.PL DIR= \ MAKEFILE=$(MAKE_APERL_FILE) LINKTYPE=static \ - MAKEAPERL=1 NORECURS=1 CCCDLFLAGS= + MAKEAPERL=1 NORECURS=1 CCCDLFLAGS= \ + PREFIX=C:/Temp # --- MakeMaker test section: End of Patch.

Subject:

Makefile.PL

use ExtUtils::MakeMaker; open FH, '>Foo.pm'; print FH "package Foo;\n1;\n"; close FH; WriteMakefile(NAME => 'Foo', INSTALLPRIVLIB => '$(INSTALLARCHLIB)');

Fri Nov 20 00:40:49 2009 mschwern [...] cpan.org - Correspondence added

INSTALLPRIVLIB => '$(INSTALLARCHLIB)' Well there's your problem! I could spend a lot of time puzzling out why that would cause a circular definition. Or I can ask what that's trying to accomplish and see if there's a better way. Why you do that?

Fri Nov 20 00:40:50 2009 The RT System itself - Status changed from 'new' to 'open'

Fri Nov 20 20:34:41 2009 shay [...] cpan.org - Correspondence added

On Fri Nov 20 00:40:49 2009, MSCHWERN wrote: Show quoted text

> INSTALLPRIVLIB => '$(INSTALLARCHLIB)' > > Well there's your problem! > > I could spend a lot of time puzzling out why that would cause a circular > definition. Or I can ask what that's trying to accomplish and see if > there's a better way. Why you do that?

It's a long story. I'll try to summarize it: IO-Compress used to have XS code, but was changed to pure-Perl. That meant that when upgrading IO-Compress in an architecture-dependent perl installation the new files would get installed into lib and wouldn't be seen if there was an older IO-Compress already installed in the archlib. To get around that, IO-Compress's Makefile.PL tries to install files into archlib, which it originally did like this (in version 2.021): INST_LIB => 'blib/arch' That worked fine on Unix, but on Win32 it went wrong with some forward-slash/back-slash confusion and ended up installing two sets of files: one into lib and one into archlib. The problem was caused by the ExtUtils::Install command-line wanting to install blib/arch into lib and blib\arch into archlib. I noted that when INSTALLDIRS is set to 'perl' (as it is in this case) files are copied from INST_LIB to INSTALLPRIVLIB, so it is really INSTALLPRIVLIB (the target of the copy) rather than INST_LIB (the source of the copy) that we want to manipulate, so I suggested using INSTALLPRIVLIB => '$(INSTALLARCHLIB)' instead. That worked perfectly (in version 2.022), until someone found that it didn't work with PREFIX. I pointed out that using INSTALL_BASE instead still worked perfectly well, and is recommended over PREFIX by the EU-MM docs anyway, but that doesn't help the bug reporter who has a bunch of scripts for CPAN testing that all use PREFIX. You can see the problem with the original INST_LIB => 'blib/arch' solution by changing the Makefile.PL in this bug report to: use ExtUtils::MakeMaker; open FH, '>Foo.pm'; print FH "package Foo;\n1;\n"; close FH; WriteMakefile(NAME => 'Foo', INST_LIB => 'blib/arch'); On my architecture-dependent perl installation, that creates a Makefile containing: INST_ARCHLIB = blib\arch INST_LIB = blib/arch INSTALLDIRS = site DESTDIR = INSTALLSITELIB = C:\perl510\site\lib DESTINSTALLSITELIB = $(DESTDIR)$(INSTALLSITELIB) INSTALLARCHLIB = C:\perl510\lib\MSWin32-x86-multi-thread DESTINSTALLARCHLIB = $(DESTDIR)$(INSTALLARCHLIB) install :: pure_install doc_install $(NOECHO) $(NOOP) pure_install :: pure_$(INSTALLDIRS)_install $(NOECHO) $(NOOP) pure_site_install :: all $(NOECHO) $(MOD_INSTALL) \ read $(SITEARCHEXP)\auto\$(FULLEXT)\.packlist \ write $(DESTINSTALLSITEARCH)\auto\$(FULLEXT)\.packlist \ $(INST_LIB) $(DESTINSTALLSITELIB) \ $(INST_ARCHLIB) $(DESTINSTALLSITEARCH) \ $(INST_BIN) $(DESTINSTALLSITEBIN) \ $(INST_SCRIPT) $(DESTINSTALLSITESCRIPT) \ $(INST_MAN1DIR) $(DESTINSTALLSITEMAN1DIR) \ $(INST_MAN3DIR) $(DESTINSTALLSITEMAN3DIR) $(NOECHO) $(WARN_IF_OLD_PACKLIST) \ $(PERL_ARCHLIB)\auto\$(FULLEXT) so the 'install' target ends up running: C:\perl510\bin\MSWin32-x86-multi-thread\perl.exe -MExtUtils::Install -e "install([ from_to => {@ARGV}, verbose => '0', uninstall_shadows => '0', dir_mode => '755' ]);" -- read C:\perl510\site\lib\MSWin32-x86-multi-thread\auto\Foo\.packlist write C:\perl510\site\lib\MSWin32-x86-multi-thread\auto\Foo\.packlist blib/arch C:\perl510\site\lib blib\arch C:\perl510\site\lib\MSWin32-x86-multi-thread blib\bin C:\perl510\bin\MSWin32-x86-multi-thread blib\script C:\perl510\bin blib\man1 C:\perl510\man\man1 blib\man3 C:\perl510\man\man3 in which you can see blib/arch will go into C:\perl510\site\lib and blib\arch will go into C:\perl510\site\lib\MSWin32-x86-multi-thread, so I end up with two copies of the .pm files. Can you suggest some other way of fixing that problem that avoids the PREFIX problem with the current INSTALLPRIVLIB => '$(INSTALLARCHLIB)' solution? -- There is more detail on the IO-Compress changes in the p5p threads that start at these points: http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-09/msg00280.html http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-10/msg00014.html http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-10/msg00095.html http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-10/msg00507.html http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2009-11/msg00486.html

Wed Apr 06 07:03:28 2011 peter.john.acklam [...] gmail.com - Correspondence added

Is there any progress on this ticket? I can't install IO-Compress-2.033.tar.gz with ExtUtils::MakeMaker v6.56 because of this bug. The "make" step fails with Makefile:89: *** Recursive variable `INSTALLARCHLIB' references itself (eventually). Stop. PMQS/IO-Compress-2.033.tar.gz /usr/bin/make install -- NOT OK

Wed Apr 06 07:31:54 2011 mschwern [...] cpan.org - Correspondence added

On Wed Apr 06 07:03:28 2011, pjacklam wrote: Show quoted text

> Is there any progress on this ticket? I can't install > IO-Compress-2.033.tar.gz with ExtUtils::MakeMaker v6.56 because of this > bug. The "make" step fails with > > Makefile:89: *** Recursive variable `INSTALLARCHLIB' references itself > (eventually). Stop. > PMQS/IO-Compress-2.033.tar.gz > /usr/bin/make install -- NOT OK

No work has been done. I haven't taken the time to pick through the analysis to figure out what's actually wrong, if anything. The work around is to use INSTALL_BASE rather than PREFIX. INSTALL_BASE is preferred in general over PREFIX anyway.

Thu Jul 31 21:54:00 2014 ETJ [...] cpan.org - Correspondence added

I claim this is obsolete since this looks like deprecated behaviour.

Thu Aug 07 10:18:17 2014 LEONT [...] cpan.org - Correspondence added

On Fri Nov 20 20:34:41 2009, SHAY wrote: Show quoted text

> On Fri Nov 20 00:40:49 2009, MSCHWERN wrote:

> > INSTALLPRIVLIB => '$(INSTALLARCHLIB)' > > > > Well there's your problem! > > > > I could spend a lot of time puzzling out why that would cause a circular > > definition. Or I can ask what that's trying to accomplish and see if > > there's a better way. Why you do that?

> > It's a long story. I'll try to summarize it: > > IO-Compress used to have XS code, but was changed to pure-Perl. That > meant that when upgrading IO-Compress in an architecture-dependent perl > installation the new files would get installed into lib and wouldn't be > seen if there was an older IO-Compress already installed in the archlib. > > To get around that, IO-Compress's Makefile.PL tries to install files > into archlib, which it originally did like this (in version 2.021): > > INST_LIB => 'blib/arch' > > That worked fine on Unix, but on Win32 it went wrong with some > forward-slash/back-slash confusion and ended up installing two sets of > files: one into lib and one into archlib. The problem was caused by the > ExtUtils::Install command-line wanting to install blib/arch into lib and > blib\arch into archlib.

I think it would have been more sensible to set it to «INST_LIB => '$(INST_ARCH)'», as that will always contain the platform appropriate path, but that would still be fragile because it depends on an implementation ordering detail. Show quoted text

> I noted that when INSTALLDIRS is set to 'perl' (as it is in this case) > files are copied from INST_LIB to INSTALLPRIVLIB, so it is really > INSTALLPRIVLIB (the target of the copy) rather than INST_LIB (the source > of the copy) that we want to manipulate, so I suggested using > > INSTALLPRIVLIB => '$(INSTALLARCHLIB)' > > instead. > > That worked perfectly (in version 2.022), until someone found that it > didn't work with PREFIX. I pointed out that using INSTALL_BASE instead > still worked perfectly well, and is recommended over PREFIX by the EU-MM > docs anyway, but that doesn't help the bug reporter who has a bunch of > scripts for CPAN testing that all use PREFIX.

The number of people who truly grok how PREFIX works in MakeMaker is likely to be 0. You shouldn't override INSTALL* values lightly. Leon

Mon Dec 08 18:21:14 2014 ETJ [...] cpan.org - Correspondence added

Could someone boil this down to a simple test case that still fails? If so, I'll cook up a test and fix it.

Thu Dec 11 04:00:36 2014 shay [...] cpan.org - Correspondence added

On Mon Dec 08 18:21:14 2014, ETJ wrote: Show quoted text

> Could someone boil this down to a simple test case that still fails? > If so, I'll cook up a test and fix it.

There are two problems here: 1. The problem with using INSTALLPRIVLIB => '$(INSTALLARCHLIB)' still remains and is easily reproduced with this script: use ExtUtils::MakeMaker; open FH, '>Foo.pm'; print FH "package Foo;\n1;\n"; close FH; WriteMakefile(NAME => 'Foo', INSTALLPRIVLIB => '$(INSTALLARCHLIB)'); If you run "perl Makefile.PL" then all is well, but if you run "perl Makefile.PL PREFIX=C:/Temp" then the Makefile that gets written contains a recursive variable definition: INSTALLARCHLIB = $(INSTALLARCHLIB) (When run without PREFIX=C:/Temp on the command-line, the Makefile contains something like: INSTALLARCHLIB = C:\perl5\lib which is obviously fine.) 2. The problem with using INST_LIB => 'blib/arch' has now mostly gone away because INSTALLARCHLIB now comes out the same as INSTALLSITELIB, so blib/arch gets copied to the same destination as blib\arch, although it is arguably still wrong that the files are getting copied twice. You can reproduce that problem with this script: use ExtUtils::MakeMaker; open FH, '>Foo.pm'; print FH "package Foo;\n1;\n"; close FH; WriteMakefile(NAME => 'Foo', INST_LIB => 'blib/arch'); When you run "perl Makefile.PL" followed by "nmake install" you will see something like: Installing C:\perl5\site\lib\Foo.pm Installing C:\perl5\site\lib\Foo.pm Appending installation info to C:\perl5\lib/perllocal.pod i.e. Foo.pm gets installed twice. That's because pure_site_install is: $(NOECHO) $(MOD_INSTALL) \ read "$(SITEARCHEXP)\auto\$(FULLEXT)\.packlist" \ write "$(DESTINSTALLSITEARCH)\auto\$(FULLEXT)\.packlist" \ "$(INST_LIB)" "$(DESTINSTALLSITELIB)" \ "$(INST_ARCHLIB)" "$(DESTINSTALLSITEARCH)" \ "$(INST_BIN)" "$(DESTINSTALLSITEBIN)" \ "$(INST_SCRIPT)" "$(DESTINSTALLSITESCRIPT)" \ "$(INST_MAN1DIR)" "$(DESTINSTALLSITEMAN1DIR)" \ "$(INST_MAN3DIR)" "$(DESTINSTALLSITEMAN3DIR)" $(NOECHO) $(WARN_IF_OLD_PACKLIST) \ "$(PERL_ARCHLIB)\auto\$(FULLEXT)" where $(INST_LIB) is blib/arch and $(INST_ARCHLIB) is blib\arch, but $(DESTINSTALLSITELIB) and $(DESTINSTALLSITEARCH) are both C:\perl5\site\lib (In my original report, $(DESTINSTALLSITELIB) and $(DESTINSTALLSITEARCH) were different, so Foo.pm got installed twice in different locations.)