Bug #30084 for CPAN-Reporter: Test failure reported as PASS

Thu Oct 18 04:17:22 2007 SREZIC [...] cpan.org - Ticket created

Subject:

Test failure reported as PASS

Just seen while testing IO::Tee on a MSWin32 system: the test has one test failure, but CPAN::Reporter thinks that the tests passes. See here for a similar report (mine, which uses CPAN::Reporter 1.02, is not yet on the cpantesters site, as it seems): http://www.nntp.perl.org/group/perl.cpan.testers/2007/09/msg617415.html I am not sure whose fault this is. The test file in IO::Tee does not use any of the Test:: modules, but just does "ok" and "not ok" printing itself. Therefore no special exit code is set (like it is done when using Test::*), but I can't remember that setting the exit code was required in the old days. Maybe it's ExtUtils::MakeMaker fault because it does not run the test file through Test::Harness, as it seems. Here a sample test rule generated by ExtUtils::MakeMaker on a Unix system with a single test.pl file: test_dynamic :: pure_all PERL_DL_NONLAZY=1 $(FULLPERLRUN) "-I$(INST_LIB)" -I$(INST_ARCHLIB)" $(TEST_FILE) And here the test rule when using t/*.t: test_dynamic :: pure_all PERL_DL_NONLAZY=1 $(FULLPERLRUN) "-MExtUtils::Command::MM" "-e" "test_harness($(TEST_VERBOSE), '$(INST_LIB)', '$(INST_ARCHLIB)')" $(TEST_FILES) Only the latter seems to use the Test::Harness. Regards, Slaven

Thu Oct 18 08:04:59 2007 dagolden [...] cpan.org - Correspondence added

Thank you for reporting this. However, this is a problem with IO::Tee, not CPAN::Reporter. IO::Tee provides only a single test.pl file. ExtUtils::MakeMaker runs that by itself -- not through Test::Harness. (By contrast, Module::Build runs even test.pl files through the harness -- which might be a "bug" since it's a big change from EU::MM. This is actually why there is t/54_test_report_split.t test in CPAN::Reporter -- for cases where EU:MM and M::B give different answers on a distribution.) In this case, test.pl finishes with an exit code of 0. From the perspective of an external program, that's success. It can print whatever it wants, ok, not ok, hello world, etc. The only thing that matters is the exit code. If it happened to use Test::More, Test::More has an END block that overrides the exit value to signal whether tests passed or not. In this case, IO::Tee writes TAP "by hand" and doesn't set an exit code appropriately. Thus -- it's an IO::Tee bug.

Thu Oct 18 08:05:11 2007 The RT System itself - Status changed from 'new' to 'open'

Thu Oct 18 08:05:17 2007 dagolden [...] cpan.org - Status changed from 'open' to 'rejected'

Thu Oct 18 15:44:18 2007 slaven [...] rezic.de - Correspondence added

Subject:	Re: [rt.cpan.org #30084] Test failure reported as PASS
Date:	18 Oct 2007 21:41:38 +0200
To:	bug-CPAN-Reporter [...] rt.cpan.org
From:	Slaven Rezic <slaven [...] rezic.de>

" via RT" <bug-CPAN-Reporter@rt.cpan.org> writes: Show quoted text

> <URL: http://rt.cpan.org/Ticket/Display.html?id=30084 > > > Thank you for reporting this. > > However, this is a problem with IO::Tee, not CPAN::Reporter. IO::Tee > provides only a single test.pl file. ExtUtils::MakeMaker runs that by > itself -- not through Test::Harness. > > (By contrast, Module::Build runs even test.pl files through the harness > -- which might be a "bug" since it's a big change from EU::MM. This is > actually why there is t/54_test_report_split.t test in CPAN::Reporter -- > for cases where EU:MM and M::B give different answers on a distribution.) > > In this case, test.pl finishes with an exit code of 0. From the > perspective of an external program, that's success. It can print > whatever it wants, ok, not ok, hello world, etc. The only thing that > matters is the exit code. > > If it happened to use Test::More, Test::More has an END block that > overrides the exit value to signal whether tests passed or not. In this > case, IO::Tee writes TAP "by hand" and doesn't set an exit code > appropriately. Thus -- it's an IO::Tee bug. >

I am not sure. If I create a module stub using h2xs from an old perl distribution (say, 5.00505), then it creates only a test.pl file and it looks like this: __BEGIN__ # Before `make install' is performed this script should be runnable with # `make test'. After `make install' it should work as `perl test.pl' ######################### We start with some black magic to print on failure. # Change 1..1 below to 1..last_test_to_print . # (It may become useful if the test is moved to ./t subdirectory.) BEGIN { $| = 1; print "1..1\n"; } END {print "not ok 1\n" unless $loaded;} use Foo; $loaded = 1; print "ok 1\n"; ######################### End of black magic. # Insert your test code below (better if it prints "ok 13" # (correspondingly "not ok 13") depending on the success of chunk 13 # of the test code): __END__ No "exit 1" in case of errors here. I also cannot remember ever to put an "exit ..." line into my hand-written test files (before Test::* module were available). So my opinion is that EUMM should run the test.pl also through Test::Harness. Regards, Slaven -- Slaven Rezic - slaven <at> rezic <dot> de babybike - routeplanner for cyclists in Berlin handheld (e.g. Compaq iPAQ with Linux) version of bbbike http://bbbike.sourceforge.net

Thu Oct 18 15:44:28 2007 The RT System itself - Status changed from 'rejected' to 'open'

Thu Oct 25 12:04:03 2007 dagolden [...] cpan.org - Correspondence added

From:

DAGOLDEN [...] cpan.org

On Thu Oct 18 15:44:18 2007, slaven@rezic.de wrote: Show quoted text

> No "exit 1" in case of errors here. I also cannot remember ever to put > an "exit ..." line into my hand-written test files (before Test::* > module were available). So my opinion is that EUMM should run the > test.pl also through Test::Harness.

That's an EU::MM issue, not a CPAN::Reporter issue. Here is the result of installing this on MSWin32 with CPAN::Reporter turned *off*: [snip] ok 25 not ok 26 ok 27 KENSHAN/IO-Tee-0.64.tar.gz C:\strawberry-perl\dmake\bin\dmake.EXE test -- OK Running make install Prepending C:\Temp\cpan\build\IO-Tee-0.64-Ya0DLi/blib/arch C:\Temp\cpan\build\IO -Tee-0.64-Ya0DLi/blib/lib to PERL5LIB for 'install' Installing C:\strawberry-perl\perl\site\lib\IO\Tee.pm Writing c:\strawberry-perl\perl\site\lib\auto\IO\Tee\.packlist Appending installation info to c:\strawberry-perl\perl\lib/perllocal.pod KENSHAN/IO-Tee-0.64.tar.gz C:\strawberry-perl\dmake\bin\dmake.EXE install UNINST=1 -- OK As you can see, CPAN treats the test as a success and proceeds to installation. As that is all that CPAN::Reporter is trying to assess, then the current behavior is "correct" -- even if the current behavior isn't ideal. I'm going to re-close this issue. David

Thu Oct 25 12:04:30 2007 dagolden [...] cpan.org - Status changed from 'open' to 'rejected'

Bug #30084 for CPAN-Reporter: Test failure reported as PASS

Preferred bug tracker