Bug #11718 for CPAN-Mini-Inject: CPAN::Mini::Inject::updpackages() does not update the line count

Wed Mar 02 10:40:56 2005 Guest - Ticket created

Subject:

CPAN::Mini::Inject::updpackages() does not update the line count

I've set up a local mirror and a repository for module injection. When reloading the indexes in the CPAN shell, I get the following: Show quoted text

cpan> reload indexes

CPAN: Storable loaded ok Going to read /home/jwest/.cpan/Metadata Database was generated on Wed, 02 Mar 2005 13:40:25 GMT CPAN: LWP::UserAgent loaded ok Fetching with LWP: ftp://localhost/pub/CPAN/authors/01mailrc.txt.gz Going to read /home/jwest/.cpan/sources/authors/01mailrc.txt.gz CPAN: Compress::Zlib loaded ok Fetching with LWP: ftp://localhost/pub/CPAN/modules/02packages.details.txt.gz Going to read /home/jwest/.cpan/sources/modules/02packages.details.txt.gz Warning: Your /home/jwest/.cpan/sources/modules/02packages.details.txt.gz contains a Line-Count header of 30482 but I see 30483 lines there. Please check the validity of the index file by comparing it to more than one CPAN mirror. I'll continue but problems seem likely to happen. Database was generated on Wed, 02 Mar 2005 13:40:25 GMT Fetching with LWP: ftp://localhost/pub/CPAN/modules/03modlist.data.gz Going to read /home/jwest/.cpan/sources/modules/03modlist.data.gz Going to write /home/jwest/.cpan/Metadata Show quoted text

cpan>

I'm only injecting one module currently, so this discrepancy is presumably caused by the injection. Similarly, the next time I run mcpani --update and the module gets re-injected the line count increases by one. For example: jwest@reanimator(0):~$ mcpani --update -v Testing site: ftp://mirrors.rcn.net/pub/lang/CPAN/ ftp://mirrors.rcn.net/pub/lang/CPAN/ selected. authors/01mailrc.txt.gz ... up to date modules/02packages.details.txt.gz ... up to date modules/03modlist.data.gz ... up to date Injecting modules from /home/jwest/.myCPAN /home/ftp/pub/CPAN/authors/id/C/CO/CORPTECH/CTI-Autobuild-Disk-0.08.tar.gz ... injected jwest@reanimator(0):~$ And when the indexes are reloaded in the CPAN shell: Show quoted text

cpan> reload indexes

CPAN: Storable loaded ok Going to read /home/jwest/.cpan/Metadata Database was generated on Wed, 02 Mar 2005 13:40:25 GMT CPAN: LWP::UserAgent loaded ok Fetching with LWP: ftp://localhost/pub/CPAN/authors/01mailrc.txt.gz Going to read /home/jwest/.cpan/sources/authors/01mailrc.txt.gz CPAN: Compress::Zlib loaded ok Fetching with LWP: ftp://localhost/pub/CPAN/modules/02packages.details.txt.gz Going to read /home/jwest/.cpan/sources/modules/02packages.details.txt.gz Warning: Your /home/jwest/.cpan/sources/modules/02packages.details.txt.gz contains a Line-Count header of 30482 but I see 30484 lines there. Please check the validity of the index file by comparing it to more than one CPAN mirror. I'll continue but problems seem likely to happen. Database was generated on Wed, 02 Mar 2005 13:40:25 GMT Fetching with LWP: ftp://localhost/pub/CPAN/modules/03modlist.data.gz Going to read /home/jwest/.cpan/sources/modules/03modlist.data.gz Going to write /home/jwest/.cpan/Metadata Show quoted text

cpan>

No additional modules were added to be injected. Upon inspecting the 02packages.details.txt.gz either downloaded by the shell or in my local CPAN mirror I see two entries for my injected module: jwest@reanimator(0):~$ zgrep Autobuild .cpan/sources/modules/02packages.details.txt.gz CTI::Autobuild::Disk 0.08 C/CO/CORPTECH/CTI-Autobuild-Disk-0.08.tar.gz CTI::Autobuild::Disk 0.08 C/CO/CORPTECH/CTI-Autobuild-Disk-0.08.tar.gz jwest@reanimator(0):~$ Hopefully, I'm correct in my guess that these two bugs are related. If not, I apologise for not submitting them seperately. I'm running the latest and greatest of CPAN (1.7601), CPAN::Mini (0.36) and CPAN::Mini::Inject (0.16) under perl, v5.8.4 built for i386-linux-thread-multi. Please don't hesitate to ask if there are more details I can provide. Thanks! --John

Fri Apr 01 13:50:22 2005 Guest - Correspondence added

From:

Douglas K. Fischer (fischerdk AT fidoki.com)

[guest - Wed Mar 2 10:40:56 2005]: Show quoted text

> I'm only injecting one module currently, so this discrepancy is > presumably caused by the injection.

Correct. Show quoted text

> Similarly, the next time I run mcpani --update and the module gets re- > injected the line count increases by one.

Also correct. Show quoted text

> Hopefully, I'm correct in my guess that these two bugs are related.

Yes and no. Same routine, different code issues. Issue #1: Local modules re-injected with each running of mcpani --inject. This is caused by a bug in updpackages() in which the local module entry ($modules[0]) is string compared with entries in the package index file ($_). The problem is that $_ is terminated w/ a newline whereas $modules[0] isn't. Therefore the equality comparison will never be true and the local module will be re-injected. Issue #2: The Line-Count: header in the package index is not updated upon adding module entries to the file. While not critical, it does annoy the CPAN shell, and CPAN::Mini. This is caused by writing out the headers immediately upon reading them in updpackages() and not waiting until we know how many package lines there are to that "Line-Count:" can be properly updated. The patch below solves both of these problems: Doug ------------------------------------------------ diff -Pur CPAN-Mini-Inject-0.16.orig/lib/CPAN/Mini/Inject.pm CPAN-Mini-Inject-0.16/lib/CPAN/Mini/Inject.pm --- CPAN-Mini-Inject-0.16.orig/lib/CPAN/Mini/Inject.pm Sat Jan 8 22:28:46 2005 +++ CPAN-Mini-Inject-0.16/lib/CPAN/Mini/Inject.pm Fri Apr 1 13:39:37 2005 @@ -369,6 +369,8 @@ my $self=shift; my @modules=sort(@{$self->{modulelist}}); + my @headers=(); + my %modules=(); my $cpanpackages=$self->_cfg('local').'/modules/02packages.details.txt.gz'; my $newpackages=$self->_cfg('repository').'/02packages.details.txt.gz'; @@ -376,27 +378,35 @@ or croak "Cannot open local 02packages.details.txt.gz: $gzerrno"; my $inheader=1; - my $gzwrite = gzopen($newpackages,'wb') - or croak "Cannot open repository 02packages.details.txt.gz: $gzerrno"; while($gzread->gzreadline($_)) { if($inheader) { $inheader=0 unless(/\S/); - $gzwrite->gzwrite($_); + push @headers, $_; next; } + $modules{$_} = 1; + } + $gzread->gzclose; - if(defined($modules[0]) && lc($modules[0]) lt lc($_)) { - $gzwrite->gzwrite($modules[0]."\n"); - shift(@modules); - redo; + foreach my $module (@modules) { + if(defined($module)) { + $modules{$module."\n"} = 1; } - if(defined($modules[0]) && lc($modules[0]) eq lc($_)) { - shift(@modules); - next; + } + + my $pkgcount=scalar(keys %modules); + my $gzwrite = gzopen($newpackages,'wb') + or croak "Cannot open repository 02packages.details.txt.gz: $gzerrno"; + foreach my $hdr (@headers) { + if($hdr =~ /Line-Count:/) { + $hdr =~ s/\d+/$pkgcount/; } - $gzwrite->gzwrite($_); + $gzwrite->gzwrite($hdr); } - $gzread->gzclose; + foreach my $module (sort keys %modules) { + $gzwrite->gzwrite($module); + } + $gzwrite->gzclose; copy($newpackages,$cpanpackages); $self->_updperms($cpanpackages); --------------------------------------------------

Fri Apr 01 14:28:59 2005 SSORICHE [...] cpan.org - Correspondence added

RT-Send-CC:

fischerdk [...] fidoki.com

John & Doug Sorry about not getting back sooner. I have Line-Count working correctly in the current development version. I've been working on adding tests for the headers in 02packages file and also to update_mirror. This has taken longer than expected due to fixes in other code. New version will be released this weekend. Thanks a lot. -- Shawn Sorichetti

Fri Apr 01 14:44:34 2005 Guest - Correspondence added

[SSORICHE - Fri Apr 1 14:28:59 2005]: Show quoted text

> Sorry about not getting back sooner.

No problem, Shawn. Show quoted text

> New version will be released this weekend.

Thank you muchly. And Doug, your patch worked like a charm. Thank you, too. --John

Sun Apr 03 15:11:48 2005 SSORICHE [...] cpan.org - Status changed from 'new' to 'resolved'