Skip Menu |

This queue is for tickets about the CPAN-Site CPAN distribution.

Report information
The Basics
Id: 33196
Status: resolved
Worked: 1.5 hours (90 min)
Priority: 0/
Queue: CPAN-Site

People
Owner: Nobody in particular
Requestors: matisse [...] spamcop.net
Cc:
AdminCc:

Bug Information
Severity: Normal
Broken in: 0.17
Fixed in: (no value)



Subject: CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
CPAN::Site::Index::cpan_index() is incorrectly registering packages that come from .pm files that are not actually "provided" by a distro. Example: http://search.cpan.org/~marcel/Test-Compile-0.08/ contains the following: http://search.cpan.org/src/MARCEL/Test-Compile-0.08/inc/Test/Pod/Coverage.pm and that file contains these two lines: package Test::Pod::Coverage; our $VERSION = "1.08"; this causes CPAN::Site::Index::cpan_index() to register that package as "belonging" to Test-Compile-0.08 which is not really true.
I may have a patch for this - I need to get permission from $work to post it first.
Subject: Re: [rt.cpan.org #33196] CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
Date: Tue, 12 Feb 2008 22:22:53 +0100
To: Matisse Enzer via RT <bug-CPAN-Site [...] rt.cpan.org>
From: Mark Overmeer <mark [...] overmeer.net>
* Matisse Enzer via RT (bug-CPAN-Site@rt.cpan.org) [080212 20:37]: Show quoted text
> Tue Feb 12 15:37:35 2008: Request 33196 was acted upon. > Transaction: Ticket created by MATISSE > Queue: CPAN-Site > Subject: CPAN::Site::Index::cpan_index() is registering packages that are > not provided by a distro > > CPAN::Site::Index::cpan_index() is incorrectly registering packages that > come from .pm files that are not actually "provided" by a distro. > > Example: > http://search.cpan.org/~marcel/Test-Compile-0.08/ > > contains the following: > http://search.cpan.org/src/MARCEL/Test-Compile-0.08/inc/Test/Pod/Coverage.pm > > and that file contains these two lines: > > package Test::Pod::Coverage; > our $VERSION = "1.08";
cpan-site takes the official module index from the real CPAN, and mergers this with the modules found in your local set-up. From your description, I do not get an indication that this Test-Compile module is on you own system. I know that the normal CPAN indexer is quite blunt as well: it does detect that you try to reuse a package which is already used by someone else, and gives an error upon it. Even if you use .pm's in your tests. So: is this a CPAN::Site problem? -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions drs Mark A.C.J. Overmeer MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
On Tue Feb 12 21:50:24 2008, Mark@Overmeer.net wrote: Show quoted text
> cpan-site takes the official module index from the real CPAN, and > mergers this with the modules found in your local set-up. From your > description, I do not get an indication that this Test-Compile module > is on you own system.
My fault - Test-Compile is in fact in our local, mini-cpan, and we are using CPAN::Site::Index to index that mini-cpan. Show quoted text
> I know that the normal CPAN indexer is quite blunt as well: it does > detect that you try to reuse a package which is already used by > someone > else, and gives an error upon it. Even if you use .pm's in your > tests.
I'm guessing that the normal CPAN indexer has some way of ignoring .pm files that are not in a lib/ directory or at the top-level of the distro (this is the approach my patch takes.) Show quoted text
> So: is this a CPAN::Site problem?
I think it's fair to change CPAN::Site to not index .pm files that are not in the distros' lib/ directory or at the top level of the distro - does that seem reasonable?
Subject: Re: [rt.cpan.org #33196] CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
Date: Wed, 13 Feb 2008 09:19:54 +0100
To: Matisse Enzer via RT <bug-CPAN-Site [...] rt.cpan.org>
From: Mark Overmeer <solutions [...] overmeer.net>
* Matisse Enzer via RT (bug-CPAN-Site@rt.cpan.org) [080213 04:03]: Show quoted text
> My fault - Test-Compile is in fact in our local, mini-cpan, and we are > using CPAN::Site::Index to index that mini-cpan.
ok. Show quoted text
> I'm guessing that the normal CPAN indexer has some way of ignoring .pm > files that are not in a lib/ directory or at the top-level of the distro > (this is the approach my patch takes.)
It seems to be fixed, not too long ago. I had some test packages myself, which were index... but on the moment, they are not anymore. Show quoted text
> > So: is this a CPAN::Site problem?
> > I think it's fair to change CPAN::Site to not index .pm files that are > not in the distros' lib/ directory or at the top level of the distro - > does that seem reasonable?
There is no rule where to put installed files in the directory tree, so it is not that easy... although it is clear which directories to exclude. For performance reasons, CPAN::Site is bluntly scanning the tar file for lines which start with 'package'. Of course, a better implementation is possible, which usually involve the slow unpacking of the tar... -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
From: matisse [...] spamcop.net
I finally got approval this morning from $work to post the patch I made (attached to this comment.) We are using it at work and it fixes the problem.
--- Index.pm.dist 2008-01-31 12:26:30.000000000 -0800 +++ Index.pm 2008-02-12 15:12:27.000000000 -0800 @@ -3,12 +3,11 @@ # See the manual pages for details on the licensing terms. # Pod stripped from pm file by OODoc 1.03. +package CPAN::Site::Index; use warnings; use strict; -package CPAN::Site::Index; -use vars '$VERSION'; -$VERSION = '0.17'; +our $VERSION = '0.172'; use base 'Exporter'; our @EXPORT_OK = qw/cpan_index/; @@ -125,8 +124,10 @@ { my ($package, $version, $dist) = @_; print "reg(@_)\n" if $debug; + my $registered_version = $findpkgs->{$package}[0] || ''; + return if exists $findpkgs->{$package} - && $findpkgs->{$package}[0] ge $version; + && $registered_version ge $version; $findpkgs->{$package} = [ $version, $dist ]; } @@ -160,6 +161,8 @@ my $in_buf = ''; my $out_buf = ''; my $readme_fh; + my $tarball_name = basename $dist; + my ($dist_name) = ( $tarball_name =~ / (.*) \.tar\.gz /x ); BLOCK: while ($fh->sysread($in_buf, 512)) @@ -167,6 +170,12 @@ if($in_buf =~ /^(\S*?)\0/) { $file = $1; + if ( ! length $file ) { + next BLOCK; + } + if ( $file !~ /^$dist_name/ ) { + next BLOCK; + } # when the package contains non-text files, this produces garbage # print "file=$file\n" if $debug && length $file; @@ -177,8 +186,7 @@ # my $outputfn = File::Spec->catfile($File::Find::dir, $readme_file); # $outputfn =~ s/\bREADME$/\.readme/; - my $readmefn = basename $dist; - $readmefn =~ s/\.tar\.gz/\.readme/; + my $readmefn = $dist_name . '.readme'; my $outputfn = File::Spec->catfile($File::Find::dir, $readmefn); print "README full path '$outputfn'\n" if $debug; @@ -206,6 +214,7 @@ while ($out_buf =~ s/^([^\n]*)\n//) { local $_ = $1; + local $VERSION; if( m/^\s* package \s* ((\w+\:\:)*\w+) \s* ;/x ) { $package = $1; print "package=$package\n" if $debug; @@ -214,13 +223,41 @@ { $version = eval "my \$v = $1"; print "version=$version\n" if $debug; - register $package, $version, $dist - if $file && $file =~ m/\.pm$/ && $package; + if ( ok_to_register($package,$file) ) + { + register $package, $version, $dist; + } } } } } +sub ok_to_register($$) +{ + my ($package,$path) = @_; + + if ( ! $path ) + { + return; + } + + if ( $path !~ m/\.pm$/ ) + { + return; + } + + if ( ! $package ) { + return; + } + + if ( ! path_is_in_lib_or_at_top_level($path) ) + { + return; + } + + return 1; +} + sub merge_core_cpan($$$) { my ($cpan, $pkgs, $bigcpan_url) = @_; @@ -267,12 +304,12 @@ my $module = __PACKAGE__; $fh->print (<<__HEADER); File: 02packages.details.txt -URL: file:$details +URL: file://$details Description: Packages listed in CPAN and local repository Columns: package name, version, path Intended-For: Standard CPAN with additional private resources Line-Count: $lines -Written-By: $program with $module $VERSION ($how) +Written-By: $program with $module $CPAN::Site::Index::VERSION ($how) Last-Updated: $date __HEADER @@ -376,4 +413,23 @@ $ftp->close; } +sub path_is_in_lib_or_at_top_level($) +{ + my ($path) = @_; + + my @path_parts = File::Spec->splitdir($path); + if ( ! defined $path_parts[0] ) { + warn "no parts in '$path'"; + } + my $second_thing_in_path = $path_parts[1] || ''; + if ( $second_thing_in_path eq 'lib' ) { + # path is in lib directory + return 1; + } + if ( @path_parts == 2 ) { + # path is at top-level of distro + return 1; + } + return; +} 1;
I realize that you may want to use different rules for deciding what package names to register. In the patch the new subroutine: ok_to_register() handles that logic, so that could be changed to use different rules, for example, excluding files in the t/ directory.
Subject: Re: [rt.cpan.org #33196] CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
Date: Tue, 19 Feb 2008 23:50:00 +0100
To: Matisse Enzer via RT <bug-CPAN-Site [...] rt.cpan.org>
From: Mark Overmeer <solutions [...] overmeer.net>
* Matisse Enzer via RT (bug-CPAN-Site@rt.cpan.org) [080219 17:15]: Show quoted text
> Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=33196 > > We are using it at work and it fixes the problem.
+package CPAN::Site::Index; use warnings; use strict; -package CPAN::Site::Index; warnings and strict have file scope, so you can safely start a file with it. -use vars '$VERSION'; -$VERSION = '0.17'; +our $VERSION = '0.172'; Is this change really needed? local $VERSION also works for "our", isn't it? I rewrote your logic into my coding-style. It seems logical. -- Thanks for your contribution! MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
Subject: Re: [rt.cpan.org #33196] CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
Date: Tue, 19 Feb 2008 17:02:55 -0800
To: bug-CPAN-Site [...] rt.cpan.org
From: Matisse Enzer <matisse [...] matisse.net>
On Feb 19, 2008, at 2:50 PM, Mark Overmeer via RT wrote: Show quoted text
> +package CPAN::Site::Index; > use warnings; > use strict; > -package CPAN::Site::Index; > > warnings and strict have file scope, so you can safely start a > file with it. > > -use vars '$VERSION'; > -$VERSION = '0.17'; > +our $VERSION = '0.172'; > > Is this change really needed? local $VERSION also works for "our", > isn't it?
"our" is different from "local" (i'm pretty sure you know that!) - it declares the variable as a package variable, so in this case our $VERSION = '0.111'; means: $CPAN::Site::Index::VERSION = '0.111'; which is bit different from what 'use vars' does. One of the changes I made was this: -Written-By: $program with $module $VERSION ($how) +Written-By: $program with $module $CPAN::Site::Index::VERSION ($how) the way I did it there is less chance of the wrong value being used for the version. It's really not a big deal at all and i am not offended in any way if you ignore this part :-) Show quoted text
> I rewrote your logic into my coding-style. It seems logical.
No problem! I tried to copy the coding style that was present but obviously I ended up introducing a different style and it is best to keep only one style per file! -M ------------------------------------------------------- Matisse Enzer <matisse@matisse.net> http://www.matisse.net/ - http://www.eigenstate.net/
Subject: Re: [rt.cpan.org #33196] CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
Date: Wed, 20 Feb 2008 08:57:57 +0100
To: Matisse Enzer via RT <bug-CPAN-Site [...] rt.cpan.org>
From: Mark Overmeer <mark [...] overmeer.net>
* Matisse Enzer via RT (bug-CPAN-Site@rt.cpan.org) [080220 01:04]: Show quoted text
> > -use vars '$VERSION'; > > -$VERSION = '0.17'; > > +our $VERSION = '0.172'; > > > > Is this change really needed? local $VERSION also works for "our", > > isn't it?
> > "our" is different from "local" (i'm pretty sure you know that!) - it > declares the variable as a package variable, so in this case > > our $VERSION = '0.111'; > means: > $CPAN::Site::Index::VERSION = '0.111'; > > which is bit different from what 'use vars' does.
AFAIK, there is a difference between "our" and "use vars", which came up last week by a posting on perl5-porters: the our declares a global variable with file-scope, not package scope. However, for processing, there is no difference: it is just an as valid way of avoiding an error by strictness... and nicer. An additional "problem" is that the $VERSION is added by OODoc, when the packages are processed to create the distribution man-pages and code. I got fed-up maintaining those, so they are automatically taken from the Makefile.PL Show quoted text
> One of the changes I made was this: > -Written-By: $program with $module $VERSION ($how) > +Written-By: $program with $module $CPAN::Site::Index::VERSION ($how) > > the way I did it there is less chance of the wrong value being used > for the version.
Although at compile-time is determined which package is used to take the $VERSION from, there are too many "versions" in the context and therefore this clarifies the code. -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
Subject: Re: [rt.cpan.org #33196] CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
Date: Wed, 20 Feb 2008 08:52:18 -0800
To: bug-CPAN-Site [...] rt.cpan.org
From: Matisse Enzer <matisse [...] matisse.net>
On Feb 19, 2008, at 11:58 PM, Mark Overmeer via RT wrote: Show quoted text
> AFAIK, there is a difference between "our" and "use vars", which came > up last week by a posting on perl5-porters: the our declares a global > variable with file-scope, not package scope.
I believe this is incorrect - "our" declares in package-scope, "my" (and "local") are lexical. Consider the following two small files: #---- File ModuleA.pm ----- package Foo; use strict; use warnings; our $A = 'hello from package ' . __PACKAGE__; #------------------------------------------------------------------------------- package Bar; our $A = 'another hello from package ' . __PACKAGE__; #------------------------------------------------------------------------------- package Baz; our $A = 'and a final hello from package ' . __PACKAGE__; #------------------------------------------------------------------------------- 1; #--- File test_our.pl ---- use strict; use warnings; use ModuleA; print "Foo: $Foo::A\n"; print "Bar: $Bar::A\n"; print "Baz: $Baz::A\n"; exit; #-------- The run the command line: perl test_our.pl and you get: Foo: hello from package Foo Bar: another hello from package Bar Baz: and a final hello from package Baz
Subject: Re: [rt.cpan.org #33196] CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
Date: Wed, 20 Feb 2008 20:45:34 +0100
To: Matisse Enzer via RT <bug-CPAN-Site [...] rt.cpan.org>
From: Mark Overmeer <mark [...] overmeer.net>
* Matisse Enzer via RT (bug-CPAN-Site@rt.cpan.org) [080220 16:53]: Show quoted text
> I believe this is incorrect - "our" declares in package-scope, > "my" (and "local") are lexical.
Exactly as I expect. Try this: :-) use strict; use warnings; package Foo; our $A = 'hello from '. __PACKAGE__; my $C = '1'; package Bar; $A = 'final hello from package ' . __PACKAGE__; $B = 'aap'; # comment this to get it to work, see effect our $A package main; print "Foo: $Foo::A\n"; print "Bar: $Bar::A\n"; print "C=$C\n"; -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
Subject: Re: [rt.cpan.org #33196] CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
Date: Wed, 20 Feb 2008 20:11:50 -0800
To: bug-CPAN-Site [...] rt.cpan.org
From: Matisse Enzer <matisse [...] matisse.net>
On Feb 20, 2008, at 1:36 PM, Mark Overmeer via RT wrote: Show quoted text
> > Exactly as I expect. > Try this: :-) > > use strict; > use warnings; > > package Foo; > > our $A = 'hello from '. __PACKAGE__; > my $C = '1'; > > package Bar; > $A = 'final hello from package ' . __PACKAGE__; > $B = 'aap'; # comment this to get it to work, see effect our $A > > package main; > print "Foo: $Foo::A\n"; > print "Bar: $Bar::A\n"; > print "C=$C\n";
OK - I see what you mean - the line: $A = 'final hello from package ' . __PACKAGE__; is using the same $A declared with 'our' earlier, so you are right, the 'our' doesn't change the lexical scope of $A, it simply puts it in the Foo:: namespace (unlike $B which is not in the Bar:: namespace.) ------------------------------------------------------- Matisse Enzer <matisse@matisse.net> http://www.matisse.net/ - http://www.eigenstate.net/
From: matisse [...] spamcop.net
Any idea when 0.18 will be released? I want to demo our build system and would like to do it using an "official" CPAN::Site distro instead of our local monkey-patched version.
As you wish: I have just uploaded 0.18. Hopefully without bugs, because I didn't have the chance to use it.
From: matisse [...] spamcop.net
On Wed Mar 19 13:38:32 2008, MARKOV wrote: Show quoted text
> As you wish: I have just uploaded 0.18. Hopefully without bugs, because > I didn't have the chance to use it.
Looks like a tiny bug snuck in. Attached patch fixes it
--- /usr/local/CPAN/build/CPAN-Site-0.18-o0XBBH/lib/CPAN/Site/Index.pm 2008-03-19 10:35:45.000000000 -0700 +++ /Library/Perl/5.8.8/CPAN/Site/Index.pm 2008-03-20 08:00:27.000000000 -0700 @@ -194,7 +194,7 @@ # my $outputfn = File::Spec->catfile($File::Find::dir,$readme_file); # $outputfn =~ s/\bREADME$/\.readme/; - my $readmefn = "$distname.readme"; + my $readmefn = "$dist_name.readme"; my $outputfn = File::Spec->catfile($File::Find::dir, $readmefn); print "README full path '$outputfn'\n" if $debug;
Subject: Re: [rt.cpan.org #33196] CPAN::Site::Index::cpan_index() is registering packages that are not provided by a distro
Date: Fri, 21 Mar 2008 08:15:12 +0100
To: Matisse Enzer via RT <bug-CPAN-Site [...] rt.cpan.org>
From: Mark Overmeer <mark [...] overmeer.net>
* Matisse Enzer via RT (bug-CPAN-Site@rt.cpan.org) [080320 15:06]: Show quoted text
> > Queue: CPAN-Site > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=33196 > > > On Wed Mar 19 13:38:32 2008, MARKOV wrote:
> > As you wish: I have just uploaded 0.18. Hopefully without bugs, because > > I didn't have the chance to use it.
> > Looks like a tiny bug snuck in. > Attached patch fixes it
Released as 0.20 -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net