Bug #19437 for CPANPLUS: does not detect FAILed tests?

Fri Oct 21 11:40:00 2005 BARBIE [...] cpan.org - Ticket #15195: Ticket created

Subject:

UNKNOWN report are being marked as PASSes

There is a bug in the latest CPANPLUS which affects reports. Currently UNKNOWN reports are not getting marked as such, but as PASS reports, due to a missing line in Dist::MM.pm. Patch attached as applied to the latest devel tarball (15th October 2005).

--- devel/lib/CPANPLUS/Dist/MM.pm Fri Oct 14 13:50:22 2005 +++ devel-barbie/lib/CPANPLUS/Dist/MM.pm Fri Oct 21 16:30:46 2005 @@ -565,6 +565,7 @@ ### pick up on this if ( NO_TESTS_DEFINED->( $captured ) ) { msg( NO_TESTS_DEFINED->( $captured ), 0 ) + $fail++; } else { msg( loc( "MAKE TEST passed: %2", $captured ) ); }

Fri Oct 21 11:44:41 2005 BARBIE [...] cpan.org - Ticket #15195: Correspondence added

From:

barbie [...] missbarbell.co.uk

missing piece in the patch, this is the one to look at.

--- devel/lib/CPANPLUS/Dist/MM.pm Fri Oct 14 13:50:22 2005 +++ devel-barbie/lib/CPANPLUS/Dist/MM.pm Fri Oct 21 16:42:54 2005 @@ -565,6 +565,7 @@ ### pick up on this if ( NO_TESTS_DEFINED->( $captured ) ) { msg( NO_TESTS_DEFINED->( $captured ), 0 ) + $fail++; last RUN; } else { msg( loc( "MAKE TEST passed: %2", $captured ) ); }

Fri Oct 21 12:34:13 2005 BARBIE [...] cpan.org - Ticket #15195: Correspondence added

From:

barbie [...] missbarbell.co.uk

One day I'll remember to run tests before submitting a patch! OOps! This IS the correct patch :) I've looked to writing tests for this patch, but it looks a bit tricky. If you have any pointers, please let me know.

--- devel/lib/CPANPLUS/Dist/MM.pm Fri Oct 14 13:50:22 2005 +++ devel-barbie/lib/CPANPLUS/Dist/MM.pm Fri Oct 21 17:18:54 2005 @@ -564,7 +564,8 @@ ### log this occasion non-verbosely, so our test reporter can ### pick up on this if ( NO_TESTS_DEFINED->( $captured ) ) { - msg( NO_TESTS_DEFINED->( $captured ), 0 ) + msg( NO_TESTS_DEFINED->( $captured ), 0 ); + $fail++; last RUN; } else { msg( loc( "MAKE TEST passed: %2", $captured ) ); }

Tue Oct 25 10:14:38 2005 kane [...] cpan.org - Ticket #15195: Correspondence added

[BARBIE - Fri Oct 21 12:34:13 2005]: Show quoted text

> One day I'll remember to run tests before submitting a patch! OOps! > This > IS the correct patch :) > > I've looked to writing tests for this patch, but it looks a bit > tricky. > If you have any pointers, please let me know.

--- devel/lib/CPANPLUS/Dist/MM.pm Fri Oct 14 13:50:22 2005 +++ devel-barbie/lib/CPANPLUS/Dist/MM.pm Fri Oct 21 17:18:54 2005 @@ -564,7 +564,8 @@ ### log this occasion non-verbosely, so our test reporter can ### pick up on this if ( NO_TESTS_DEFINED->( $captured ) ) { - msg( NO_TESTS_DEFINED->( $captured ), 0 ) + msg( NO_TESTS_DEFINED->( $captured ), 0 ); + $fail++; last RUN; } else { msg( loc( "MAKE TEST passed: %2", $captured ) ); } This patch looks quite Wrong -- setting $fail++ indicates that the whole phase failed, which is definately not true, just because the user didn't supply any tests. Looks more likely the logic in Report.pm needs to be altered.

Tue Oct 25 10:14:39 2005 kane [...] cpan.org - Ticket #15195: Status changed from 'new' to 'open'

Wed Oct 26 06:30:29 2005 Guest - Ticket #15195: Correspondence added

From:

barbie [...] missbarbell.co.uk

The '$fail' variable is perhaps misnamed. In _send_report() several checks (including those for an UNKNOWN report) are skipped if the 'failure' flag is set to false. Several PASS reports are getting sent to cpan-testers which should really be UNKNOWN reports. The fact that the $fail variable isn't getting set means the check for UNKNOWN is getting skipped everytime. If there needs to be a change in _send_report(), then the check for UNKNOWN needs to be done before accepting that the report is a PASS. I can look into this, if you feel it has more merit being check inside _send_report().

Mon Feb 06 08:54:02 2006 kane [...] cpan.org - Ticket #15195: Status changed from 'open' to 'stalled'

Wed Apr 12 23:38:04 2006 Guest - Ticket #15195: Correspondence added

On Fri Oct 21 11:40:00 2005, BARBIE wrote: Show quoted text

> There is a bug in the latest CPANPLUS which affects reports.

Currently Show quoted text

> UNKNOWN reports are not getting marked as such, but as PASS > reports, due to a missing line in Dist::MM.pm. Patch attached as > applied to the latest devel tarball (15th October 2005).

Wed Apr 12 23:38:04 2006 The RT System itself - Ticket #15195: Status changed from 'stalled' to 'open'

Mon May 22 17:04:46 2006 jrockway [...] cpan.org - Ticket #15195: Correspondence added

On Fri Oct 21 11:40:00 2005, BARBIE wrote: Show quoted text

> There is a bug in the latest CPANPLUS which affects reports. Currently > UNKNOWN reports are not getting marked as such, but as PASS > reports, due to a missing line in Dist::MM.pm. Patch attached as > applied to the latest devel tarball (15th October 2005).

On my system, FAILs are being reported as PASSes! Show quoted text

> <cut> > # Looks like you failed 7 tests of 10. > dubious > Test returned status 7 (wstat 1792, 0x700) > DIED. FAILED tests 2-5, 7-8, 10 > Failed 7/10 tests, 30.00% okay > Failed Test Stat Wstat Total Fail List of Failed >

------------------------------------------------------------------------------- Show quoted text

> t/02static.t 7 1792 10 7 2-5 7-8 10 > Failed 1/2 test scripts. 7/11 subtests failed. > Files=2, Tests=11, 1 wallclock secs ( 0.40 cusr + 0.03 csys = 0.43 > CPU) > > [MSG] Successfully sent 'pass' report for 'Catalyst-Plugin-Static-0.10' > Module 'Catalyst::Plugin::Static' tested successfully > No errors testing all modules

This happens on my Debian GNU/Linux (unstable) box with Perl 5.8.8 and CPANPLUS 0.061, and my OS X machine with 5.8.6 and the same version of CPANPLUS. It happens in both normal batch install mode ("cpanp install foo::bar"), and test mode ("cpanp test foo::bar"). It also happens when updating modules, etc. Let me know how I can help hunt down the cause of this. Regards, -- Jonathan Rockway <jrockway@cpan.org>

Tue May 23 10:10:37 2006 MTHURN [...] cpan.org - Ticket created

Subject:

does not detect FAILed tests?

For some reason, cpanp does not recognize test failures. Even though the test_harness process reports failure, cpanp goes ahead and installs the module. I do NOT have -force turned on. This is perl 5.9.2 on Solaris 8. [MSG] Extracted 'Devel::Fail::MakeTest' to '/disk1/src/PERL/.cpanplus/5.9.2/build/Devel-Fail-MakeTest-1.005' Running [/usr/local/bin/perl5.9.2 -MCPANPLUS::Internals::Utils::Autoflush Makefile.PL ]... Writing Makefile for Devel::Fail::MakeTest Running [/usr/local/bin/make UNINST=1]... make_all_was_successful. Running [/usr/local/bin/make test UNINST=1]... PERL_DL_NONLAZY=1 /usr/local/bin/perl5.9.2 "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/fail.... # Failed test in t/fail.t at line 2. # Looks like you failed 1 test of 1. dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 1 Failed 1/1 tests, 0.00% okay Failed Test Stat Wstat Total Fail List of Failed ------------------------------------------------------------------------------- t/fail.t 1 256 1 1 1 Failed 1/1 test scripts. 1/1 subtests failed. Files=1, Tests=1, 0 wallclock secs ( 0.11 cusr + 0.01 csys = 0.12 CPU) [MSG] MAKE TEST passed: PERL_DL_NONLAZY=1 /usr/local/bin/perl5.9.2 "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/fail.... # Failed test in t/fail.t at line 2. # Looks like you failed 1 test of 1. dubious Test returned status 1 (wstat 256, 0x100) DIED. FAILED test 1 Failed 1/1 tests, 0.00% okay Failed Test Stat Wstat Total Fail List of Failed ------------------------------------------------------------------------------- t/fail.t 1 256 1 1 1 Failed 1/1 test scripts. 1/1 subtests failed. Files=1, Tests=1, 0 wallclock secs ( 0.11 cusr + 0.01 csys = 0.12 CPU) Running [/usr/local/bin/make install UNINST=1]... make: Nothing to be done for `install'. Module 'Devel::Fail::MakeTest' installed successfully CPAN Terminal> s conf allow_build_interactivity '1' base '/disk1/src/PERL/.cpanplus' buildflags 'uninst=1' cpantest '0' debug '0' dist_type '' email 'martin.thurn@ngc.com' extractdir '' fetchdir '' flush '1' force '0' lib [] makeflags 'UNINST=1' makemakerflags '' md5 '1' no_update '0' passive '1' prefer_bin '0' prefer_makefile '1' prereqs '1' shell 'CPANPLUS::Shell::Default' signature '0' skiptest '0' storable '1' timeout '5' verbose '1' -- - - Martin 'Kingpin' Thurn

Wed Jun 07 07:25:53 2006 kane [...] cpan.org - Correspondence added

Show quoted text

> # Failed test in t/fail.t at line 2. > # Looks like you failed 1 test of 1. > dubious > Test returned status 1 (wstat 256, 0x100) > DIED. FAILED test 1 > Failed 1/1 tests, 0.00% okay > Failed Test Stat Wstat Total Fail List of Failed >

------------------------------------------------------------------------------ - Show quoted text

> t/fail.t 1 256 1 1 1 > Failed 1/1 test scripts. 1/1 subtests failed. > Files=1, Tests=1, 0 wallclock secs ( 0.11 cusr + 0.01 csys = 0.12 > CPU) > [MSG] MAKE TEST passed: PERL_DL_NONLAZY=1 /usr/local/bin/perl5.9.2 "-

This can mean 1 of 3 things, which i hope you'll be able to track down, as i'm not able to reproduce this problem on my box, but i've had multiple reports of this happening: 1) Your version of EU::MM is not exiting 'make test' with a non-zero status on 'fail' (unlikely) 2) IPC::Cmd is not registering the failure of the 'make command'. This would have to be either in and older version (grab the latest and try?) or in a certain codepath. It works for me with IPC::Run enabled and using IPC::Open3 (you can enable/disable them from the shell by typing: '! $IPC::Cmd::USE_IPC_RUN = 0;' and running the test again) 3) a missing 'scalar' declaration in the conditional to check success throws off the check. Technically 'if' forces boolean context so it shouldn't matter, and i've nto seen a change with or without the 'scalar' declaration, but it's worth a try. Patch below. Hopefully this gives you what you need to track down the problem. If you have any questions, by all means ask. Thanks, ==== //depot/cpanplus/devel/lib/CPANPLUS/Dist/MM.pm#64 - /Users/kane/sources/p4/ cpanplus/devel/lib/CPANPLUS/Dist/MM.pm ==== 580c580,581 < if(run( command => [$make, 'test', $makeflags], --- Show quoted text

> if( scalar run( > command => [$make, 'test', $makeflags],

Wed Jun 07 07:26:01 2006 The RT System itself - Status changed from 'new' to 'open'

Wed Jun 07 07:26:22 2006 kane [...] cpan.org - Status changed from 'open' to 'stalled'

Wed Jun 07 07:29:11 2006 kane [...] cpan.org - Ticket #15195: Correspondence added

Show quoted text

> On my system, FAILs are being reported as PASSes!

I've had a similar report which I replied to with possible causes and debugging steps. I hope this helps you track down the issue as well: http://rt.cpan.org/Ticket/Display.html?id=19437

Wed Jun 07 07:29:54 2006 kane [...] cpan.org - Ticket #15195: Merged into ticket #19437

Wed Jun 07 07:29:54 2006 kane [...] cpan.org - Merged into ticket #19437

Thu Jun 08 10:47:17 2006 MTHURN [...] cpan.org - Correspondence added

I chased this down. The cause is bug #19505 in the Test::Harness queue, and there is a patch there to fix. http://rt.cpan.org/Ticket/Display.html?id=19505 -- - - Martin 'Kingpin' Thurn

Thu Jun 08 10:47:18 2006 The RT System itself - Status changed from 'stalled' to 'open'

Tue Jun 13 09:44:28 2006 kane [...] cpan.org - Correspondence added

On Thu Jun 08 10:47:17 2006, MTHURN wrote: Show quoted text

> I chased this down. The cause is bug #19505 in the Test::Harness > queue, and there is a patch there to fix. > > http://rt.cpan.org/Ticket/Display.html?id=19505

Fantastic! I've added a prereq on T::H 2.62 for the next release of CPANPLUS (0.070_01) which will be hitting CPAN shortly. Good work and thanks for the help!

Tue Jun 13 09:44:31 2006 kane [...] cpan.org - Status changed from 'open' to 'resolved'