Skip Menu |

This queue is for tickets about the Module-ScanDeps CPAN distribution.

Report information
The Basics
Id: 22115
Status: resolved
Priority: 0/
Queue: Module-ScanDeps

People
Owner: smueller [...] cpan.org
Requestors: jv [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in:
  • 0.50
  • 0.51
  • 0.52
  • 0.53
  • 0.54
  • 0.55
  • 0.56
  • 0.57
  • 0.58
  • 0.59
  • 0.60
  • 0.61
  • 0.63
Fixed in: (no value)



Subject: PARring Term::ReadLine includes Tk
When a PAR includes Term::ReadLine, it will include all Tk libs and modules although Term::ReadLine does not 'use' nor 'require' any Tk. E.g. pp -d -o xx.par /usr/lib/perl5/5.8.8/Term/ReadLine.pm
Thanks for the bug report! In fact, this bug is in Module::ScanDeps, not in PAR. PAR uses Module::ScanDeps to scan the code-to-be-packaged for its dependencies. The bogus dependency of Term::ReadLine on Tk must be introduced somewhere in Term::ReadLine. These days, I find myself maintainer of Module::ScanDeps as well (I recall saying "I'll make a release to fix this bug I found"), so I'll move the ticket into its queue if RT will let me. Steffen
RT-Send-CC: audreyt [...] audreyt.org
[Sending a CC to Audrey because I don't know whether she reads the tickets of this queue. Audrey: What would you do in this case?] Hi Johan, Okay, I traced the bug. I am unsure how to fix it, though. Term::ReadLine actually has something like "Tk-compatibility code" in it. Line 369 reads: $count_DoOne++, Tk::DoOneEvent(0) until $giveup; That line is incorrectly considered to make use of Tk. In fact, Module::ScanDeps is rather liberal with false positives because it discards all false positives which do not exist on the system as a module/file. One of the heuristics, in fact around line 634 of ScanDeps.pm reads: return $1 if /\b(\w[\w:]*)::\w+\(/; That picks up any strings of the form "Foo::Bar::baz(" and takes "Foo::Bar" as the namespace of a module to require. The crudeness of the approach takes little from the overall scarily high success rate. Except that if you consider code that is only executed ever if and only if that specific module was loaded, it blows up. I don't know how to proceed. It would be easy to add a statement akin to "if this is Term::ReadLine, I'm scanning, don't pick up any Tk dependencies!", but that's a bad hack stacked on top of another. I would hope Audrey could chime in on this. The only really satisfying method to do this is to rewrite Module::ScanDeps to use PPI. But I can't do that. I hope this clears things up a little. Steffen
Subject: Re: [rt.cpan.org #22115] PARring Term::ReadLine includes Tk
Date: Mon, 16 Oct 2006 09:59:02 +0200
To: bug-Module-ScanDeps [...] rt.cpan.org
From: jv [...] cpan.org (Johan Vromans)
[Quoting Steffen Müller via RT, on October 15 2006, 16:59, in "[rt.cpan.org #22115]"] Show quoted text
> return $1 if /\b(\w[\w:]*)::\w+\(/; > > That picks up any strings of the form "Foo::Bar::baz(" and takes > "Foo::Bar" as the namespace of a module to require.
Why? I'd say only 'use Foo::Bar' or 'require Foo::Bar' validates the inclusion of module Foo::Bar. If a module uses Foo::Bar without including it, there must be some other module that includes Foo::Bar, otherwise we are dealing with an application that is inconsitent. E.g. File Foo.pm: package Foo; use Foo::Bar; ... File Foo/Bar.pm package Foo::Bar; ... File Blech.pm: package Blech; use Foo; my $x = new Foo::Bar; ... Module Blech depends on Foo, but not on Foo::Bar. It does, indirectly via Foo, but not directly. Show quoted text
> I don't know how to proceed. It would be easy to add a statement > akin to "if this is Term::ReadLine, I'm scanning, don't pick up any > Tk dependencies!", but that's a bad hack stacked on top of another. > I would hope Audrey could chime in on this.
For this particular case, I'd be happy if I could override the inclusion of Tk. Unfortunately, the command line switch -X Tk does not have this effect. Either Module::ScanDeps or PAR needs a more extensive way to control what gets included (and excluded) anyway. PAR.yaml, maybe? Show quoted text
> I hope this clears things up a little.
It does, thanks for that. BTW: Are there any known quirks with the processing of INIT blocks? I have an app that loads a particular module and starts the main code. When run, it show the following debugging messages: == (module BEGIN block execute) == (module code execures) == (module INIT block executes) ... application runs ... However, when PARred, I get: == (module BEGIN block execute) == (module code execures) ... application runs ... == (module INIT block executes) It's a big application so it is not easy to track this down. But maybe there are some known issues that can help me finding (and fixing) it. Thanks, BTW2: (Sorry, can't stop) I always install modules in a private library. But despite my CPAN::MyConfig settings PAR insists on writing to the system perl library. Is this a Module::Install versus MakeMaker thing? I'd be willing to make an issue for it, but I'd like to check it first. -- Johan
RT-Send-CC: audreyt [...] audreyt.org
On Mo. 16. Okt. 2006, 03:59:33, JV wrote: Show quoted text
> [Quoting Steffen Müller via RT, on October 15 2006, 16:59, in > "[rt.cpan.org #22115]"]
> > return $1 if /\b(\w[\w:]*)::\w+\(/; > > > > That picks up any strings of the form "Foo::Bar::baz(" and takes > > "Foo::Bar" as the namespace of a module to require.
> > Why? I'd say only 'use Foo::Bar' or 'require Foo::Bar' validates the > inclusion of module Foo::Bar. If a module uses Foo::Bar without > including it, there must be some other module that includes Foo::Bar, > otherwise we are dealing with an application that is inconsitent.
I'm not going to defend Module::ScanDeps' approach because neither did I craft nor implement it. But I'll point out that you're missing something here: Perl programmers have been very, very good at creating devices to defeat anything that tries to parse portions of the code. How about this: use Scary::Device; Scary::Device->LoadPlugins(qw/Scary::Device::Plugin::Foo/); And trust me: This is just *one* of the cases relying on "use" and "require" fails. I think the idea behind M::SD's heuristics was laid on that foundation. It picks up anything that might be a module and sees whether that module exists in the library. If it does, it assumes it found a valid dependency. It's bad that way, but users are generally much more annoyed if a dependency is *not* picked up than if a dependency is picked up too much. In case of Tk, however, that's a *lot* that's picked up. Again, in parts because Tk has the weirdest auto-loading mechanisms of them all! Adding a quick workaround to Module::ScanDeps' scan_line or scan_deps routines would be possible. I can do that within a couple of days. Rewriting ScanDeps with a better approach is for somebody else. [...] Show quoted text
> Module Blech depends on Foo, but not on Foo::Bar. It does, indirectly > via Foo, but not directly.
I understand that. Show quoted text
> > I don't know how to proceed. It would be easy to add a statement > > akin to "if this is Term::ReadLine, I'm scanning, don't pick up any > > Tk dependencies!", but that's a bad hack stacked on top of another. > > I would hope Audrey could chime in on this.
> > For this particular case, I'd be happy if I could override the > inclusion of Tk. Unfortunately, the command line switch -X Tk does not > have this effect. > > Either Module::ScanDeps or PAR needs a more extensive way to control > what gets included (and excluded) anyway. PAR.yaml, maybe?
Hmm. I was going to suggest -X Tk. Perhaps -X Tk actually excludes it but not its dependencies (which are found to be Tk::*)? I just checked it and that seems to be the case. Why? My hunch is that pp's -X excludes a specific module *after* the scanning has taken place. That means: The tree generated while scanning isn't pruned while scanning but a node is removed afterwards. I agree that this is the wrong behaviour. I am unsure how difficult it would be to implement the (more) correct behaviour. For the lack of time, I can't do it now. I cannot make any promises either. In general, though it won't help here, you can put any series of command line arguments into a text file and do pp @textfile to use it as the source for parameters if the list is too long. Show quoted text
> BTW: Are there any known quirks with the processing of INIT blocks? I > have an app that loads a particular module and starts the main code. > When run, it show the following debugging messages: > > == (module BEGIN block execute) > == (module code execures) > == (module INIT block executes) > ... application runs ... > > However, when PARred, I get: > > == (module BEGIN block execute) > == (module code execures) > ... application runs ... > == (module INIT block executes) > > It's a big application so it is not easy to track this down. But maybe > there are some known issues that can help me finding (and fixing) it.
IIRC, this is absolutely an issue. Have a look at the TODO list, specifically the last item of the original list: http://par.wikia.com/wiki/DevelopmentInProgress It talks about check blocks, but IIRC, that applies to INIT blocks as well. I do not claim to understand the issue, though. You will have to try to get Audrey involved for that. I have been bitten by this before. I think some of the attribute modules use check/init blocks. Good luck tracking it down! Show quoted text
> BTW2: (Sorry, can't stop) I always install modules in a private > library. But despite my CPAN::MyConfig settings PAR insists on writing > to the system perl library. Is this a Module::Install versus MakeMaker > thing? I'd be willing to make an issue for it, but I'd like to check > it first.
Perhaps the Module::Install auto-install feature does that? I.e. it asks whether it should install Module::Foo now (before PAR) because it's missing for feature Bar? If you say yes to that, it's outside CPAN because it's done by the bundled Module::Install. I think that is the issue. Adam Kennedy has been hacking on the PAR Makefile.PL a bit to remove the M::I auto-installation and feature stuff in favour of a sane build process. I would hope this last problem goes away when that is released. (Still chasing him so he commits it.) Steffen
Subject: Re: [rt.cpan.org #22115] PARring Term::ReadLine includes Tk
Date: Mon, 16 Oct 2006 11:12:22 +0200
To: bug-Module-ScanDeps [...] rt.cpan.org
From: jv [...] cpan.org (Johan Vromans)
[Quoting Steffen Müller via RT, on October 16 2006, 04:24, in "[rt.cpan.org #22115]"] Show quoted text
> use Scary::Device; > Scary::Device->LoadPlugins(qw/Scary::Device::Plugin::Foo/); > > And trust me: This is just *one* of the cases relying on "use" and > "require" fails.
I'd say that's exactly why PAR provides a -M option. I use such constructs myself, and I always specify the necessary modules with -M. No pain. Show quoted text
> I think the idea behind M::SD's heuristics was laid on that foundation. > It picks up anything that might be a module and sees whether that module > exists in the library. If it does, it assumes it found a valid > dependency.
Hmm. So if I remove Tk from the perl library, my 'problem' would be over? Show quoted text
> > BTW: Are there any known quirks with the processing of INIT blocks?
> IIRC, this is absolutely an issue. Have a look at the TODO list, > specifically the last item of the original list: > http://par.wikia.com/wiki/DevelopmentInProgress
I see, thanks. Show quoted text
> Perhaps the Module::Install auto-install feature does that?
I stand corrected. I retried, and all went well. However: % perl -MCPAN -e "install qw(PAR)" CPAN: Storable loaded ok Going to read /home/jv/.cpan/Metadata Database was generated on Sun, 01 Oct 2006 09:40:30 GMT Running install for module PAR Running make for S/SM/SMUELLER/PAR-0.954.tar.gz CPAN: LWP::UserAgent loaded ok Fetching with LWP: ftp://ftp.squirrel.nl/pub/CPAN/authors/id/S/SM/SMUELLER/PAR-0.954.tar.gz LWP failed with code[404] message[File 'PAR-0.954.tar.gz' not found] ... Someone thinks 0.954 is most recent (but it is not on CPAN). 0.956 is on CPAN, but not in the index? Puzzled... Thanks for your time, patience and nice work, Johan
Subject: Re: [rt.cpan.org #22115] PARring Term::ReadLine includes Tk
Date: Mon, 16 Oct 2006 12:51:56 +0200
To: bug-Module-ScanDeps [...] rt.cpan.org
From: jv [...] cpan.org (Johan Vromans)
[Quoting Steffen Müller via RT, on October 16 2006, 04:24, in "[rt.cpan.org #22115]"] Show quoted text
> Hmm. I was going to suggest -X Tk. Perhaps -X Tk actually excludes it > but not its dependencies (which are found to be Tk::*)? I just checked > it and that seems to be the case. Why? My hunch is that pp's -X excludes > a specific module *after* the scanning has taken place. That means: The > tree generated while scanning isn't pruned while scanning but a node is > removed afterwards.
From this, I would expect the Tk modules to disappear if I specify each of them explicitly with -X ? This doesn't seem to happen. -- Johan
On Mo. 16. Okt. 2006, 06:53:06, JV wrote: Show quoted text
> From this, I would expect the Tk modules to disappear if I specify > each of them explicitly with -X ? This doesn't seem to happen.
Funny. I can't test it right now, but Tk.pm was omitted when I specified -X Tk. Perhaps the dll's aren't omitted. I didn't check that. Steffen
On Mo. 16. Okt. 2006, 05:12:51, JV wrote: Show quoted text
> [Quoting Steffen Müller via RT, on October 16 2006, 04:24, in > "[rt.cpan.org #22115]"]
> > use Scary::Device; > > Scary::Device->LoadPlugins(qw/Scary::Device::Plugin::Foo/); > > > > And trust me: This is just *one* of the cases relying on "use" and > > "require" fails.
> > I'd say that's exactly why PAR provides a -M option. > I use such constructs myself, and I always specify the necessary > modules with -M. No pain.
Alternatively, adding a "require Foo::Bar" to the program works well. Show quoted text
> > I think the idea behind M::SD's heuristics was laid on that
> foundation.
> > It picks up anything that might be a module and sees whether that
> module
> > exists in the library. If it does, it assumes it found a valid > > dependency.
> > Hmm. So if I remove Tk from the perl library, my 'problem' would be > over?
Yes. I can't say I like it that way. I was a little shocked when I discovered that. Short of a Module::ScanDeps rewrite, there's nothing that can be done in this specific regard. Show quoted text
> % perl -MCPAN -e "install qw(PAR)" > CPAN: Storable loaded ok > Going to read /home/jv/.cpan/Metadata > Database was generated on Sun, 01 Oct 2006 09:40:30 GMT > Running install for module PAR > Running make for S/SM/SMUELLER/PAR-0.954.tar.gz > CPAN: LWP::UserAgent loaded ok > Fetching with LWP: > ftp://ftp.squirrel.nl/pub/CPAN/authors/id/S/SM/SMUELLER/PAR- > 0.954.tar.gz > LWP failed with code[404] message[File 'PAR-0.954.tar.gz' not > found] > ... > > Someone thinks 0.954 is most recent (but it is not on CPAN). 0.956 is > on CPAN, but not in the index? Puzzled...
I get this: (for "m par") DESCRIPTION Perl Archive Toolkit CPAN_USERID AUTRIJUS (Audrey Tang <cpan@audreyt.org>) CPAN_VERSION 0.956 CPAN_FILE S/SM/SMUELLER/PAR-0.956.tar.gz UPLOAD_DATE 2006-10-03 DSLIP_STATUS RmhOp (released,mailing-list,hybrid,object-oriented,Standard-Perl) MANPAGE PAR - Perl Archive Toolkit INST_FILE /usr/local/share/perl/5.8.7/PAR.pm INST_VERSION 0.956 Perhaps your mirror needs updating? PAUSE told me it indexed PAR 0.956 alright. I uploaded a 0.954, but it was a broken release. Hence, I deleted it very quickly. Steffen
Subject: Re: [rt.cpan.org #22115] PARring Term::ReadLine includes Tk
Date: Mon, 16 Oct 2006 16:36:24 +0200
To: bug-Module-ScanDeps [...] rt.cpan.org
From: jv [...] cpan.org (Johan Vromans)
[Quoting Steffen Müller via RT, on October 16 2006, 09:20, in "[rt.cpan.org #22115]"] Show quoted text
> > I'd say that's exactly why PAR provides a -M option. > > I use such constructs myself, and I always specify the necessary > > modules with -M. No pain.
> > Alternatively, adding a "require Foo::Bar" to the program works well.
This is not possible if you two conflicting modules, only one of which can be used. Show quoted text
> Perhaps your mirror needs updating? PAUSE told me it indexed PAR > 0.956 alright. I uploaded a 0.954, but it was a broken release. > Hence, I deleted it very quickly.
Apparently, one of my metadata files got currupt. It is solved now. BTW: This is what I currently use: pp -p -o xx.par \ -I lib \ -M <extra modules> \ -a <extra files> \ script/app zip -d xx.par lib/Tk\* zip -d xx.par lib/auto/Tk/\* parl -b -Oapp.par xx.par Not elegant, but it seems to do the trick. Thanks, Johan
Module::ScanDeps 0.67 is on its way to CPAN. It fixes the specific bug you witnessed but doesn't make the overall problem of fragile heuristics go away. Please allow for some time before the tarball hits your CPAN mirror. Steffen