Bug #102689 for Module-ScanDeps: Module::ScanDeps isn't effective

Wed Mar 11 09:05:28 2015 ZDM [...] cpan.org - Ticket created

Subject:

Module::ScanDeps isn't effective

Hi. Module::ScanDeps isn't effective. Perl::LibExtractor produce much less deps tree. It use different algorithm - "use package" in separate process and parse %INC after - so only really used packages are found. Also it works much more faster. It will be useful to add possibility to use Perl::LibExtractor with pp or use similar approach to build deps tree.

Wed Mar 11 09:36:01 2015 RSCHUPP [...] cpan.org - Correspondence added

Am 2015-03-11 09:05:28, ZDM schrieb: Show quoted text

> Module::ScanDeps isn't effective.

You probably meant: efficient. Show quoted text

> Perl::LibExtractor produce much less deps tree. > > It use different algorithm - "use package" in separate process and > parse %INC after - so only really used packages are found. > > Also it works much more faster.

... but it doesn't catch all dependencies either. Not that the above approach is also used by Module::ScanDeps if use the --compile or --execute option. But the results only ADD dependencies to the ones found during static analysis. Show quoted text

> It will be useful to add possibility to use Perl::LibExtractor with pp > or use similar approach to build deps tree.

Code please - if you want to contribute some, add it to a wishlist bug for PAR::Packer. Cheers, Roderich

Wed Mar 11 09:36:02 2015 The RT System itself - Status changed from 'new' to 'open'

Wed Mar 11 09:36:06 2015 RSCHUPP [...] cpan.org - Status changed from 'open' to 'rejected'

Wed Mar 11 09:41:50 2015 ether [...] cpan.org - Correspondence added

There's a few different mechanisms for finding deps... Perl::PrereqScanner is another that uses PPI to find statements like 'use' and 'require' in the code. These could possibly all be used alternately or in tandem, depending on user options.

Wed Mar 11 09:57:29 2015 dzagashev [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 11 Mar 2015 15:57:14 +0200
To:	bug-PAR-Packer [...] rt.cpan.org
From:	"dzagashev [...] gmail.com" <dzagashev [...] gmail.com>

Module::ScanDeps also didn't finds all deps, but it adds a lot of garbage, archive size is become 10 Mb vs 5 Mb if I use libextractor. In both cases I need to add deps manually. On 11.03.2015 15:36, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=102689 > > > Am 2015-03-11 09:05:28, ZDM schrieb:

>> Module::ScanDeps isn't effective.

> You probably meant: efficient. >

>> Perl::LibExtractor produce much less deps tree. >> >> It use different algorithm - "use package" in separate process and >> parse %INC after - so only really used packages are found. >> >> Also it works much more faster.

> ... but it doesn't catch all dependencies either. > > Not that the above approach is also used by Module::ScanDeps if use > the --compile or --execute option. But the results only ADD dependencies > to the ones found during static analysis. >

>> It will be useful to add possibility to use Perl::LibExtractor with pp >> or use similar approach to build deps tree.

> Code please - if you want to contribute some, add it to a wishlist bug for PAR::Packer. > > Cheers, Roderich > > >

Wed Mar 11 10:04:16 2015 dzagashev [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 11 Mar 2015 16:04:01 +0200
To:	bug-PAR-Packer [...] rt.cpan.org
From:	"dzagashev [...] gmail.com" <dzagashev [...] gmail.com>

Also option to add all packages under some namespace will be very useful (like in perlapp). For example: -M AnyEvent::* This will allow to not to enumerate lot of packages in command line. On 11.03.2015 15:36, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=102689 > > > Am 2015-03-11 09:05:28, ZDM schrieb:

>> Module::ScanDeps isn't effective.

> You probably meant: efficient. >

>> Perl::LibExtractor produce much less deps tree. >> >> It use different algorithm - "use package" in separate process and >> parse %INC after - so only really used packages are found. >> >> Also it works much more faster.

> ... but it doesn't catch all dependencies either. > > Not that the above approach is also used by Module::ScanDeps if use > the --compile or --execute option. But the results only ADD dependencies > to the ones found during static analysis. >

>> It will be useful to add possibility to use Perl::LibExtractor with pp >> or use similar approach to build deps tree.

> Code please - if you want to contribute some, add it to a wishlist bug for PAR::Packer. > > Cheers, Roderich > > >

Wed Mar 11 10:11:17 2015 RSCHUPP [...] cpan.org - Correspondence added

Am 2015-03-11 09:57:29, dzagashev@gmail.com schrieb: Show quoted text

> Module::ScanDeps also didn't finds all deps, but it adds a lot of > garbage, archive size is become 10 Mb vs 5 Mb if I use libextractor.

I strives to err on the side of completeness. Cheers, Roderich

Wed Mar 11 10:26:21 2015 RSCHUPP [...] cpan.org - Correspondence added

Am 2015-03-11 10:04:16, dzagashev@gmail.com schrieb: Show quoted text

> Also option to add all packages under some namespace will be very > useful > (like in perlapp). > > For example: -M AnyEvent::* > > This will allow to not to enumerate lot of packages in command line.

On the other hand, this will produce exactly the kind of bloat you mentioned. But I'm open to code contributions. The mechanism for finding all installed modules in a namespace is already there (sub _glob_in_inc), hence you only to enhance option parsing. On the other hand, this logic is already implemented in Module::ScanDeps, e.g. if it sees "use PPI" it will add all modules "below" PPI. For AnyEevent there's currently no such rule, but it's a onliner for me to add one - just file a bug against Module::ScanDeps. Cheers, Roderich

Wed Mar 11 10:29:19 2015 RSCHUPP [...] cpan.org - Correspondence added

Am 2015-03-11 09:41:50, ETHER schrieb: Show quoted text

> There's a few different mechanisms for finding deps... > Perl::PrereqScanner is another that uses PPI to find statements like > 'use' and 'require' in the code. These could possibly all be used > alternately or in tandem, depending on user options.

Sure, IIRC pp.pm should let you "plug in" different scanner and packer modules, though it currently knows only one of each kind. Cheers, Roderich

Wed Mar 11 10:44:10 2015 dzagashev [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 11 Mar 2015 16:43:55 +0200
To:	bug-PAR-Packer [...] rt.cpan.org
From:	"dzagashev [...] gmail.com" <dzagashev [...] gmail.com>

Understood. Will try to find time to make patch. On 11.03.2015 16:26, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=102689 > > > Am 2015-03-11 10:04:16, dzagashev@gmail.com schrieb:

>> Also option to add all packages under some namespace will be very >> useful >> (like in perlapp). >> >> For example: -M AnyEvent::* >> >> This will allow to not to enumerate lot of packages in command line.

> On the other hand, this will produce exactly the kind of bloat you > mentioned. But I'm open to code contributions. The mechanism > for finding all installed modules in a namespace is already there > (sub _glob_in_inc), hence you only to enhance option parsing. > > On the other hand, this logic is already implemented in Module::ScanDeps, > e.g. if it sees "use PPI" it will add all modules "below" PPI. > For AnyEevent there's currently no such rule, but it's a onliner for me to add one - just file a bug against Module::ScanDeps. > > Cheers, Roderich > >

Fri Mar 13 11:31:55 2015 RSCHUPP [...] cpan.org - Queue changed from PAR-Packer to Module-ScanDeps

Fri Mar 13 11:38:55 2015 RSCHUPP [...] cpan.org - Correspondence added

On 2015-03-11 10:44:10, dzagashev@gmail.com wrote: Show quoted text

> Will try to find time to make patch.

Or just try the following patch to Module::ScanDeps which should make pp automagically include anything in namespace AnyEvent::* (and their dependencies) without the need for any explicit -M AnyEvent::... --- lib/Module/ScanDeps.pm (revision 1555) +++ lib/Module/ScanDeps.pm (revision 1556) @@ -225,6 +225,7 @@ my %Preload; %Preload = ( 'AnyDBM_File.pm' => [qw( SDBM_File.pm )], + 'AnyEvent.pm' => 'sub', 'Authen/SASL.pm' => 'sub', 'B/Hooks/EndOfScope.pm' => [qw( B/Hooks/EndOfScope/PP.pm B/Hooks/EndOfScope/XS.pm )], 'Bio/AlignIO.pm' => 'sub', Cheers, Roderich

Wed Mar 18 04:58:04 2015 dzagashev [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 18 Mar 2015 10:57:50 +0200
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	"dzagashev [...] gmail.com" <dzagashev [...] gmail.com>

Hi. I have a question. I use parl to make executable. parl -B -O1.exe par.zip Parl also pack perl dll and other binary deps. Is it possible to pack all this deps with upx? This can reduce size on several Mb. Is there is any hook, that allows to do something with this files, before parl concatenate them with exe header? On 13.03.2015 17:38, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=102689 > > > On 2015-03-11 10:44:10, dzagashev@gmail.com wrote:

>> Will try to find time to make patch.

> Or just try the following patch to Module::ScanDeps > which should make pp automagically include anything in namespace AnyEvent::* > (and their dependencies) without the need for any explicit -M AnyEvent::... > > > --- lib/Module/ScanDeps.pm (revision 1555) > +++ lib/Module/ScanDeps.pm (revision 1556) > @@ -225,6 +225,7 @@ > my %Preload; > %Preload = ( > 'AnyDBM_File.pm' => [qw( SDBM_File.pm )], > + 'AnyEvent.pm' => 'sub', > 'Authen/SASL.pm' => 'sub', > 'B/Hooks/EndOfScope.pm' => [qw( B/Hooks/EndOfScope/PP.pm B/Hooks/EndOfScope/XS.pm )], > 'Bio/AlignIO.pm' => 'sub', > > > Cheers, Roderich

Wed Mar 18 06:19:26 2015 roderich.schupp [...] googlemail.com - Correspondence added

CC:	"par [...] perl.org" <par [...] perl.org>
Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 18 Mar 2015 11:19:13 +0100
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	Roderich Schupp <roderich.schupp [...] googlemail.com>

On Wed, Mar 18, 2015 at 9:58 AM, dzagashev@gmail.com via RT < bug-Module-ScanDeps@rt.cpan.org> wrote: Is it possible to pack all this deps with upx? This can reduce size on Show quoted text

> several Mb. >

IMHO that "several Mb" claim is exaggerated. The largest file by far in there is perl*.dll which is ~1.5 MB in size and typically would compress around 50%. Is there is any hook, that allows to do something with this files, Show quoted text

> before parl concatenate them with exe header?

Nope. Cheers, Roderich

Wed Mar 18 06:25:50 2015 dzagashev [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 18 Mar 2015 12:25:32 +0200
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	"dzagashev [...] gmail.com" <dzagashev [...] gmail.com>

Thanks for reply. Under strawberry perl there are 15 standard dlls (including mingw deps), that can be compressed from 3,2 to 1,3 total size. On 18.03.2015 12:19, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=102689 > > > On Wed, Mar 18, 2015 at 9:58 AM, dzagashev@gmail.com via RT < > bug-Module-ScanDeps@rt.cpan.org> wrote: > > Is it possible to pack all this deps with upx? This can reduce size on

>> several Mb. >>

> IMHO that "several Mb" claim is exaggerated. The largest file by far in > there is perl*.dll which is ~1.5 MB in size and typically would compress > around 50%. > > Is there is any hook, that allows to do something with this files,

>> before parl concatenate them with exe header?

> > Nope. > > Cheers, Roderich >

Wed Mar 18 06:46:01 2015 roderich.schupp [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 18 Mar 2015 11:45:48 +0100
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	Roderich Schupp <roderich.schupp [...] gmail.com>

On Wed, Mar 18, 2015 at 11:25 AM, dzagashev@gmail.com via RT < bug-Module-ScanDeps@rt.cpan.org> wrote: Show quoted text

> Under strawberry perl there are 15 standard dlls (including mingw deps), > that can be compressed from 3,2 to 1,3 total size. >

... but not worth the added complexity. Cheers, Roderich

Wed Mar 18 10:35:22 2015 dzagashev [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 18 Mar 2015 16:35:06 +0200
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	"dzagashev [...] gmail.com" <dzagashev [...] gmail.com>

Solution - compress dlls and then reinstall PAR::Packer. Using upx and manual deps scanner I got 4 Mb par executable instead of 11 for complex project. On 18.03.2015 12:46, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=102689 > > > On Wed, Mar 18, 2015 at 11:25 AM, dzagashev@gmail.com via RT < > bug-Module-ScanDeps@rt.cpan.org> wrote: >

>> Under strawberry perl there are 15 standard dlls (including mingw deps), >> that can be compressed from 3,2 to 1,3 total size. >>

> ... but not worth the added complexity. > > Cheers, Roderich >

Wed Mar 18 10:42:43 2015 dzagashev [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 18 Mar 2015 16:42:21 +0200
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	"dzagashev [...] gmail.com" <dzagashev [...] gmail.com>

Also, if we use rar or 7zip or something else with good compression, instead of Archive::Zip - size can be >2Mb instead of 4. On 18.03.2015 12:46, Roderich Schupp via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=102689 > > > On Wed, Mar 18, 2015 at 11:25 AM, dzagashev@gmail.com via RT < > bug-Module-ScanDeps@rt.cpan.org> wrote: >

>> Under strawberry perl there are 15 standard dlls (including mingw deps), >> that can be compressed from 3,2 to 1,3 total size. >>

> ... but not worth the added complexity. > > Cheers, Roderich >

Wed Mar 18 11:07:25 2015 roderich.schupp [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #102689] Module::ScanDeps isn't effective
Date:	Wed, 18 Mar 2015 16:07:16 +0100
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	Roderich Schupp <roderich.schupp [...] gmail.com>

On Wed, Mar 18, 2015 at 3:42 PM, dzagashev@gmail.com via RT < bug-Module-ScanDeps@rt.cpan.org> wrote: Show quoted text

> Also, if we use rar or 7zip or something else with good compression, > instead of Archive::Zip - size can be >2Mb instead of 4. >

Please stop this thread on the bug tracker - it's a BUG TRACKER, not a general discussion forum - and take it to the par@perl.org mailing list instead. Cheers, Roderich

Thu Mar 19 04:05:10 2015 SREZIC [...] cpan.org - Correspondence added

On 2015-03-18 11:07:25, roderich.schupp@gmail.com wrote: Show quoted text

> On Wed, Mar 18, 2015 at 3:42 PM, dzagashev@gmail.com via RT < > bug-Module-ScanDeps@rt.cpan.org> wrote: >

> > Also, if we use rar or 7zip or something else with good compression, > > instead of Archive::Zip - size can be >2Mb instead of 4. > >

> > Please stop this thread on the bug tracker - it's a BUG TRACKER,

RT = Request Tracker, not bug tracker. Show quoted text

> not a > general > discussion forum - and take it to the par@perl.org mailing list instead. > > Cheers, Roderich

Regards, Slaven