Bug #101569 for Module-ScanDeps: Incorrect module parsing if Moose is included

Thu Jan 15 23:59:55 2015 david.trzcinski [...] skipfault.com - Ticket created

Subject:	Incorrect module parsing if Moose is included
Date:	Thu, 15 Jan 2015 20:59:40 -0800
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	david.trzcinski [...] skipfault.com

Message body is not shown because it is too large.

Message body is not shown because sender requested not to inline it.

Fri Jan 16 05:32:32 2015 RSCHUPP [...] cpan.org - Correspondence added

On 2015-01-15 23:59:55, david.trzcinski@skipfault.com wrote: Show quoted text

> Example code attached

It's missing... I can't reproduce the problem with a simplistic scandeps -c -e 'use File::ChangeNotify;' Cheers, Roderich

Fri Jan 16 05:32:33 2015 The RT System itself - Status changed from 'new' to 'open'

Sun Jan 18 16:50:51 2015 david.trzcinski [...] skipfault.com - Correspondence added

Subject:	Re: [rt.cpan.org #101569] Incorrect module parsing if Moose is included
Date:	Sun, 18 Jan 2015 13:50:37 -0800
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	david.trzcinski [...] skipfault.com

Hi Roderich, Not sure why the attachment didn't come through correctly. I'm using the following code as a quick test : use Modern::Perl; no if $] >= 5.017011, warnings => 'experimental::smartmatch'; use Fcntl qw(:flock); use strict; use warnings; use AI::Categorizer; use AI::Categorizer::Collection::Files; use Algorithm::NaiveBayes::Model::Frequency; use File::Spec; use File::Copy; use File::ChangeNotify; use AI::Categorizer::Learner::NaiveBayes; use List::MoreUtils::PP; and the complete code is : #!/usr/bin/perl use Modern::Perl; no warnings; no if $] >= 5.017011, warnings => 'experimental::smartmatch'; # do singleton check and possible bailout as early as possible. use Fcntl qw(:flock); my $lockfile = 'E:\\FlowData\\PreProcess\\classifier.lock'; sub BailOut { print "$0 is already running. Exiting.\n"; exit(1); } open( my $fhpid, '>>', $lockfile ) or die "error: open '$lockfile': $!"; flock( $fhpid, LOCK_EX | LOCK_NB ) or BailOut(); use strict; no warnings; use AI::Categorizer; use AI::Categorizer::Collection::Files; use Algorithm::NaiveBayes::Model::Frequency; use File::Spec; use File::Copy; use File::ChangeNotify; our $cat_file = 'E:\\FlowData\\PreProcess\\cats.txt'; our $path = 'E:\\FlowData\\ALL'; my $watcher = File::ChangeNotify->instantiate_watcher( directories => [$path], filter => qr/\.(?:xml)$/, ); use AI::Categorizer::Learner::NaiveBayes; our %files; our $nb = AI::Categorizer::Learner::NaiveBayes->restore_state('E:\\FlowData\\PreProcess\\state'); print "Begining to watch $path for changes.\n"; while ( my @events = $watcher->wait_for_events() ) { print "New files have been detected.\n"; open my $out_fh, '>', $cat_file; opendir( DIR, $path ) or die $!; while ( my $file = readdir(DIR) ) { # Use a regular expression to ignore files beginning with a period next if ( $file =~ m/^\./ ); print {$out_fh} "$file \n"; } closedir(DIR); close $out_fh; my $c = new AI::Categorizer::Collection::Files( path => $path, category_file => $cat_file ); while ( my $document = $c->next ) { my $hypothesis = $nb->categorize($document); print "Classified ", $hypothesis->document_name(); print " as ", $hypothesis->best_category, "\n"; my $original = $path . "\\" . $hypothesis->document_name(); my $classified = 'E:\\FlowData\\PreProcess\\'.$hypothesis->best_category().'\\'.$hypothesis->document_name(); if ( $original =~ /^(.*)$/ ) { $original = $1; # $data now untainted } else { die "Bad data in $original"; # log this somewhere } if ( $classified =~ /^(.*)$/ ) { $classified = $1; # $data now untainted } else { die "Bad data in $classified"; # log this somewhere } move( $original, $classified ); } } Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=101569 > > > On 2015-01-15 23:59:55, david.trzcinski@skipfault.com wrote:

>> Example code attached

> > It's missing... > I can't reproduce the problem with a simplistic > > scandeps -c -e 'use File::ChangeNotify;' > > > Cheers, Roderich >

Sun Jan 18 17:48:59 2015 david.trzcinski [...] skipfault.com - Correspondence added

Subject:	Re: [rt.cpan.org #101569] Incorrect module parsing if Moose is included
Date:	Sun, 18 Jan 2015 14:48:45 -0800
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	david.trzcinski [...] skipfault.com

Strangely, your simplistic test also fails on my machine : Show quoted text

PS>scandeps -c -e 'use File::ChangeNotify;'

# Use of runtime loader module Module::Runtime detected. Results of static scanning may be incomplete. # Use of runtime loader module Module::Implementation detected. Results of static scanning may be incomplete. (set by Moose): No such file or directory at C:/perl/site/lib/Module/ScanDeps/DataFeed.pm line 26. CHECK failed--call queue aborted. SYSTEM ERROR in compiling C:\Users\DTRZCI~1\AppData\Local\Temp\gVaoBef3eG: 512 at C:/perl/site/lib/Module/ScanDeps.pm line 1348. For some additional information, my environment and Module::ScanDeps version is as follows : PS C:\Users\dtrzcinski\workspace\Clear Pre Processing> perl -V Summary of my perl5 (revision 5 version 20 subversion 1) configuration: Platform: osname=MSWin32, osvers=6.1, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef useithreads=define, usemultiplicity=define use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DNDEBUG -DWIN32 -D_CONSOLE -DNO_STRICT -DPERL_TEXTMODE_SCRIPTS -DPE RL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -D_USE_32BIT_TIME_T', optimize='-O1 -MD -Zi -DNDEBUG', cppflags='-DWIN32' ccversion='18.00.31101', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -libpath:"c:\perl\lib\CORE" -machine:x86' libpth=\lib libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib ol eaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt. lib perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.li b oleaut32.lib netapi32.lib uuid.lib ws2_32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msv crt.lib libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl520.lib gnulibc_version='' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf -libpath:"c:\perl\lib\CORE" -machine:x86 ' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS PERL_MALLOC_WRAP PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF Built under MSWin32 Compiled at Jan 14 2015 14:56:04 @INC: C:/perl/site/lib/MSWin32-x86-multi-thread C:/perl/site/lib C:/perl/lib . PS C:\Users\dtrzcinski\workspace\Clear Pre Processing> cpan -D Module::ScanDeps Colorize_output is set to true but Win32::Console::ANSI is not installed. To activate colorized output, please install Win32::Console::ANSI. CPAN: Storable loaded ok (v2.51) Reading 'C:\Perl\cpan\Metadata' Database was generated on Sun, 18 Jan 2015 10:53:18 GMT Module::ScanDeps ------------------------------------------------------------------------- CPAN: Module::CoreList loaded ok (v5.20141220) (no description) R/RS/RSCHUPP/Module-ScanDeps-1.17.tar.gz C:\perl\site\lib\Module\ScanDeps.pm Installed: 1.17 CPAN: 1.17 up to date Roderich Schupp (RSCHUPP) rschupp@cpan.org PS C:\Users\dtrzcinski\workspace\Clear Pre Processing> cpan -D File::ChangeNotify Colorize_output is set to true but Win32::Console::ANSI is not installed. To activate colorized output, please install Win32::Console::ANSI. CPAN: Storable loaded ok (v2.51) Reading 'C:\Perl\cpan\Metadata' Database was generated on Sun, 18 Jan 2015 10:53:18 GMT File::ChangeNotify ------------------------------------------------------------------------- CPAN: Module::CoreList loaded ok (v5.20141220) (no description) D/DR/DROLSKY/File-ChangeNotify-0.24.tar.gz C:\perl\site\lib\File\ChangeNotify.pm Installed: 0.24 CPAN: 0.24 up to date Dave Rolsky (DROLSKY) autarch@urth.org Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=101569 > > > On 2015-01-15 23:59:55, david.trzcinski@skipfault.com wrote:

>> Example code attached

> > It's missing... > I can't reproduce the problem with a simplistic > > scandeps -c -e 'use File::ChangeNotify;' > > > Cheers, Roderich >

Sun Jan 18 18:59:11 2015 RSCHUPP [...] cpan.org - Correspondence added

On 2015-01-18 17:48:59, david.trzcinski@skipfault.com wrote: Show quoted text

> Strangely, your simplistic test also fails on my machine :

Yeah, fails for me on Windows, too, but works on Linux. Cheers, Roderich

Mon Jan 19 08:27:09 2015 RSCHUPP [...] cpan.org - Correspondence added

Am 2015-01-18 18:59:11, RSCHUPP schrieb: Show quoted text

> Yeah, fails for me on Windows, too, but works on Linux.

OK, nailed it: Moose puts "fake" entries into %INC, eg. $INC{"E:\\Strawberry\\perl\\site\\lib\\Module\\ScanDeps\\DataFeed.pm"} = "(set by Moose)"; Later on Modules::ScanDeps::DataFeed innocently tries to print out the contents of %INC by (slightly simplified) while (my ($k, $v) = each %INC) { print "$k => ", Cwd::abs_path($v), "\n"; } Now here's the difference between Linux and Windows: Linux: $ perl -MCwd -E "say Cwd::abs_path('(set by Moose)'); say 'OK';" /home/roderich/(set by Moose) OK but on Windows Perl (ie. Cwd.pm) throws an exception: Show quoted text

> perl -MCwd -E "say Cwd::abs_path('(set by Moose)'); say 'OK';"

(set by Moose): No such file or directory at -e line 1. Please try the following patch for Module::ScanDeps. Cheers, Roderich --- lib/Module/ScanDeps/DataFeed.pm~ 2014-02-09 16:09:11.000000000 +0100 +++ lib/Module/ScanDeps/DataFeed.pm 2015-01-19 14:13:30.299294400 +0100 @@ -23,7 +23,7 @@ # (2) If a key in %INC was located via a CODE or ARRAY ref or # blessed object in @INC the corresponding value in %INC contains # the ref from @INC. - if (defined $v && !ref $v) + if (defined $v && !ref $v && -e $v) { $_INC{$k} = Cwd::abs_path($v); }

Mon Jan 19 16:36:13 2015 david.trzcinski [...] skipfault.com - Correspondence added

Subject:	Re: [rt.cpan.org #101569] Incorrect module parsing if Moose is included
Date:	Tue, 20 Jan 2015 07:35:49 +1000
To:	bug-Module-ScanDeps [...] rt.cpan.org
From:	David Trzcinski <david.trzcinski [...] skipfault.com>

That did the trick. Both -c and -x are now operational. On 19 January 2015 11:27:15 pm AEST, Roderich Schupp via RT <bug-Module-ScanDeps@rt.cpan.org> wrote: Show quoted text

><URL: https://rt.cpan.org/Ticket/Display.html?id=101569 > > >Am 2015-01-18 18:59:11, RSCHUPP schrieb:

>> Yeah, fails for me on Windows, too, but works on Linux.

> >OK, nailed it: >Moose puts "fake" entries into %INC, eg. > >$INC{"E:\\Strawberry\\perl\\site\\lib\\Module\\ScanDeps\\DataFeed.pm"} >= "(set by Moose)"; > >Later on Modules::ScanDeps::DataFeed innocently tries to print >out the contents of %INC by (slightly simplified) > >while (my ($k, $v) = each %INC) >{ > print "$k => ", Cwd::abs_path($v), "\n"; >} > >Now here's the difference between Linux and Windows: > >Linux: >$ perl -MCwd -E "say Cwd::abs_path('(set by Moose)'); say 'OK';" >/home/roderich/(set by Moose) >OK > >but on Windows Perl (ie. Cwd.pm) throws an exception:

>> perl -MCwd -E "say Cwd::abs_path('(set by Moose)'); say 'OK';"

>(set by Moose): No such file or directory at -e line 1. > >Please try the following patch for Module::ScanDeps. > >Cheers, Roderich > >--- lib/Module/ScanDeps/DataFeed.pm~ 2014-02-09 16:09:11.000000000 >+0100 >+++ lib/Module/ScanDeps/DataFeed.pm 2015-01-19 14:13:30.299294400 >+0100 >@@ -23,7 +23,7 @@ > # (2) If a key in %INC was located via a CODE or ARRAY ref or > # blessed object in @INC the corresponding value in %INC contains > # the ref from @INC. >- if (defined $v && !ref $v) >+ if (defined $v && !ref $v && -e $v) > { > $_INC{$k} = Cwd::abs_path($v); > } > > > > > >

Mon Jan 19 16:58:31 2015 RSCHUPP [...] cpan.org - Correspondence added

On 2015-01-19 16:36:13, david.trzcinski@skipfault.com wrote: Show quoted text

> That did the trick. Both -c and -x are now operational.

Thanks for testing. Fix is contained in Module::ScanDeps 1.18, just released. Cheers, Roderich

Mon Jan 19 16:58:34 2015 RSCHUPP [...] cpan.org - Status changed from 'open' to 'resolved'