Bug #92496 for File-Sharedir-Install: Tests Fail under parallel testing

Mon Jan 27 00:19:11 2014 KENTNL [...] cpan.org - Ticket created

Subject:

Tests Fail under parallel testing

exporting HARNESS_OPTIONS=jN for some values of N >= 2 causes tests to fail in random ways based on the order and timing of tests:

http://www.cpantesters.org/cpan/report/ad7b7236-8096-11e3-bd44-3abc9783d55a

http://www.cpantesters.org/cpan/report/c6d535e4-8304-11e3-a4b1-4a592b17323a

The usual cause for this is several tests sharing the same paths or files for intermediate state, clobbering each others generated files at odd times, or some assumed test execution order resulting in one test requiring another test to generate a file prior to its execution.

This failure is not machine or platform specific: http://matrix.cpantesters.org/?dist=File-ShareDir-Install%200.07;reports=1

ETHER is somebody I know also executes tests with parallel harness.

And testing confirms failure does not occur for me without the parallelism.

Mon Jan 27 18:04:03 2014 perl [...] pied.nu - Correspondence added

Thank you for the bug report. This module will only ever be used to by other modules at install-time. The shared path you refer to is only shared between modules/distributions. If a distribution is being installed in parallel, it could trip over itself. That said, why would one ever install a distribution in parallel? cpan and cpanm will prevent it. This leaves 2 sysadmins doing "make install" at the same moment; an unlikely scenario.

Mon Jan 27 18:04:03 2014 The RT System itself - Status changed from 'new' to 'open'

Mon Jan 27 19:27:29 2014 ether [...] cpan.org - Correspondence added

On 2014-01-27 15:04:03, GWYN wrote: Show quoted text

> Thank you for the bug report. > > This module will only ever be used to by other modules at install- > time. The shared path you refer to is only shared between > modules/distributions. If a distribution is being installed in > parallel, it could trip over itself. > > That said, why would one ever install a distribution in parallel? > cpan and cpanm will prevent it. This leaves 2 sysadmins doing "make > install" at the same moment; an unlikely scenario.

You misunderstand what is being parallelized -- it is not multiple installations, but multiple *tests from the same installation*. See what happens when you do this in the repository: prove -lr -j9 t It is quite common to run tests in parallel. I have HARNESS_OPTIONS=j9 set in my main dev environment, as I have a quad-core cpu.

Tue Jan 28 11:20:04 2014 perl [...] pied.nu - Correspondence added

On Mon Jan 27 19:27:29 2014, ETHER wrote: Show quoted text

> You misunderstand what is being parallelized -- it is not multiple > installations, but multiple *tests from the same installation*.

I understand this. I'm just wondering how this would be a problem in production. I was under the impression that parallel testing was used to flush out bugs in the module. What you are bumping into is bugs in the unit tests.

Tue Jan 28 11:23:57 2014 perl [...] pied.nu - Correspondence added

Included patch adds $$ to temporary file and directory names. This fixes this issue. If this is satisfactory, I'll push another version out. I would still like to know how running unit tests in parallel is useful.

Subject:

unique-names.01.patch

Index: t/10_makefile.t =================================================================== --- t/10_makefile.t (revision 1612) +++ t/10_makefile.t (working copy) @@ -10,11 +10,11 @@ use ExtUtils::MakeMaker; -my $FILE = 'test-Makefile'; -rmtree( [ qw( tlib troot ) ], 0, 0 ); +my $FILE = "test-$$-Makefile"; +rmtree( [ qw( tlib-$$ troot-$$ ) ], 0, 0 ); END { $FILE and -f $FILE and unlink $FILE; - rmtree( [ qw( tlib troot ) ], 0, 0 ); + rmtree( [ qw( tlib-$$ troot-$$ ) ], 0, 0 ); } use File::ShareDir::Install; @@ -29,8 +29,8 @@ WriteMakefile( NAME => 'File::ShareDir::Install', VERSION_FROM => 'lib/File/ShareDir/Install.pm', - INST_LIB => 'tlib/lib', - PREFIX => 'troot', + INST_LIB => "tlib-$$/lib", + PREFIX => "troot-$$", MAKEFILE => $FILE, PREREQ_PM => {}, ($] >= 5.005 ? @@ -59,7 +59,7 @@ ##### mysystem( $Config{make}, '-f', $FILE ); -my $TOP = "tlib/lib/auto/share"; +my $TOP = "tlib-$$/lib/auto/share"; ok( -f "$TOP/dist/File-ShareDir-Install/honk", "Copied to blib for dist" ); ok( -f "$TOP/module/My-Test/bonk", "Copied to blib for module" ); ok( -f "$TOP/module/My-Test/again", "Copied to blib for module again" ); @@ -79,7 +79,7 @@ } else { $TOP = "$1/auto/share"; - $TOP =~ s/\$$SITEPREFIX$/troot/; + $TOP =~ s/\$$SITEPREFIX$/troot-$$/; ok( -f "$TOP/dist/File-ShareDir-Install/honk", "Copied to blib for dist" ); ok( -f "$TOP/module/My-Test/bonk", "Copied to blib for module" ); ok( -f "$TOP/module/My-Test/again", "Copied to blib for module again" ); Index: t/12_delete.t =================================================================== --- t/12_delete.t (revision 1612) +++ t/12_delete.t (working copy) @@ -10,11 +10,11 @@ use ExtUtils::MakeMaker; -my $FILE = 'test-Makefile'; -rmtree( [ qw( tlib troot ) ], 0, 0 ); +my $FILE = "test-$$-Makefile"; +rmtree( [ qw( tlib-$$ troot-$$ ) ], 0, 0 ); END { $FILE and -f $FILE and unlink $FILE; - rmtree( [ qw( tlib troot ) ], 0, 0 ); + rmtree( [ qw( tlib-$$ troot-$$ ) ], 0, 0 ); } use File::ShareDir::Install; @@ -30,8 +30,8 @@ WriteMakefile( NAME => 'File::ShareDir::Install', VERSION_FROM => 'lib/File/ShareDir/Install.pm', - INST_LIB => 'tlib/lib', - PREFIX => 'troot', + INST_LIB => "tlib-$$/lib", + PREFIX => "troot-$$", MAKEFILE => $FILE, PREREQ_PM => {}, ($] >= 5.005 ? @@ -60,7 +60,7 @@ ##### mysystem( $Config{make}, '-f', $FILE ); -my $TOP = "tlib/lib/auto/share"; +my $TOP = "tlib-$$/lib/auto/share"; ok( -f "$TOP/module/My-Test/bonk", "Installed this file" ); ok( !-f "$TOP/module/My-Test/again", "Removed this file" ); ok( !-d "$TOP/dist/File-ShareDir-Install/deeper", "Removed a directory" ); Index: t/11_dotfile.t =================================================================== --- t/11_dotfile.t (revision 1612) +++ t/11_dotfile.t (working copy) @@ -10,11 +10,11 @@ use ExtUtils::MakeMaker; -my $FILE = 'test-Makefile'; -rmtree( [ qw( tlib troot ) ], 0, 0 ); +my $FILE = "test-$$-Makefile"; +rmtree( [ qw( tlib-$$ troot-$$ ) ], 0, 0 ); END { $FILE and -f $FILE and unlink $FILE; - rmtree( [ qw( tlib troot ) ], 0, 0 ); + rmtree( [ qw( tlib-$$ troot-$$ ) ], 0, 0 ); } use File::ShareDir::Install; @@ -32,8 +32,8 @@ WriteMakefile( NAME => 'File::ShareDir::Install', VERSION_FROM => 'lib/File/ShareDir/Install.pm', - INST_LIB => 'tlib/lib', - PREFIX => 'troot', + INST_LIB => "tlib-$$/lib", + PREFIX => "troot-$$", MAKEFILE => $FILE, PREREQ_PM => {}, ($] >= 5.005 ? @@ -61,7 +61,7 @@ ##### mysystem( $Config{make}, '-f', $FILE ); -my $TOP = "tlib/lib/auto/share"; +my $TOP = "tlib-$$/lib/auto/share"; ok( -f "$TOP/dist/File-ShareDir-Install/.something", "Copied a dotfile" ); ok( !-d "$TOP/dist/File-ShareDir-Install/.dir", " ... but not dotdir" ); ok( -d "$TOP/module/My-Test/.dir", "Copied a dotdir" );

Tue Jan 28 22:16:24 2014 KENTNL [...] cpan.org - Correspondence added

Its not so much a function of "useful" but a function of "Fast".

Many CPAN modules have rather complex unit tests, several hundred t/* files, and taking many many minutes to run the test suite inspires people to not run the tests at all.

So a fast optimisation that allows running NUM_CPU tests simultaneously proves advantageous.

This is is not so much an issue with things with small numbers of tests, but the feature of turning on parallel test globally is so useful in so many places it makes sense to turn it on globally.

The only downside is there are a few modules on CPAN that don't support parallel testing, and require manual intervention to install/test with parallelism turned off.

Overall, this still proves to be a performance advantage, because shaving a 10 minute test suite down to a 2 minute test suite gives you time to fix stuff in the saved time.

And we can fix the few CPAN modules that have failing tests under parallelism, and negate even that small negative =)

Wed Jan 29 10:00:59 2014 perl [...] pied.nu - Correspondence added

On Tue Jan 28 22:16:24 2014, KENTNL wrote: Show quoted text

> Its not so much a function of "useful" but a function of "Fast".

OK, this makes sense. I'll get a new version out today if I get the time. If I may, I've seen a lot of interest (3 bug reports) in this module on the last few weeks. I wrote it as a lark a while back. I'm wondering why the sudden interest.

Wed Jan 29 12:05:55 2014 KENTNL [...] cpan.org - Correspondence added

On 2014-01-30 04:00:59, GWYN wrote:
> On Tue Jan 28 22:16:24 2014, KENTNL wrote:
> > Its not so much a function of "useful" but a function of "Fast".
>
> OK, this makes sense. I'll get a new version out today if I get the
> time.
>

Thanks. Tested, And Passed :D

> If I may, I've seen a lot of interest (3 bug reports) in this module
> on the last few weeks. I wrote it as a lark a while back. I'm
> wondering why the sudden interest.

Not sure exactly, things get a little more attention when we have a growth of people testing and reporting parallel issues.

And for a very long time, the File::ShareDir approach seemed unknown and mysterious, and it has gained much traction lately, perhaps due to DZIL, and perhaps due to a few other things drinking the Kool Aid more.

And when somebody fixes something on CPAN, that tends to trigger reinstallation waves too, and reinstallation waves trigger testing, which triggers bugs being exposed and reported.

Maybe, if I'm being optimistic, the growth in bugs may be a symptom of something growing in perl itself.

Just hard to say, because the cause of the overall trend is not obvious. Like a flock of starlings, its hard to explain why an individual starling takes the path it does, or why the starlings around it do what they do. They give the illusion of an overarching intelligence, but there is none to seen. ( And this kinda verges into the question of "what is, why is, consciousness" , we give a good illusion of having a shared contiguous motive ... but at the cellular level it is just chaos )

Wed Jan 29 13:49:26 2014 perl [...] pied.nu - Correspondence added

On Wed Jan 29 12:05:55 2014, KENTNL wrote: Show quoted text

> Thanks. Tested, And Passed :D

Good. Closing ticket. Of the 3 reports, this was the only bug report. The other 2 were feature requets. I'm wondering if File::ShareDir::Install is moving to "important CPAN module" status.

Wed Jan 29 13:49:27 2014 perl [...] pied.nu - Status changed from 'open' to 'resolved'