Bug #31373 for CPAN-Reporter: Test::Harness 3.05 breaks CPAN::Reporter

Tue Dec 11 03:20:13 2007 ANDK [...] cpan.org - Ticket created

CC:	ANDYA [...] cpan.org
Subject:	Test::Harness 3.05 breaks CPAN::Reporter

Subject line says all. Andy, can you please run a CPAN::Reporter test before a release? This is not the only test failing, there are more, but please test yourself: t/52_test_report_unknown...... # Failed test 't-NoOutput: 'make test' returned 1' # at t/52_test_report_unknown.t line 69. # Failed test 't-NoOutput: 'make test' grade reported as 'unknown'' # at t/52_test_report_unknown.t line 69. # got: 'fail' # expected: 'unknown' # STDOUT: # make[4]: Entering directory `/tmp/QZ_WiyY7O7' # cp lib/Bogus/Module.pm blib/lib/Bogus/Module.pm # PERL_DL_NONLAZY=1 /home/src/perl/repoperls/installed-perls/perl/pKhaQOm/perl-5.8.0@32603/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t # t/01_Bogus...... No subtests run # # Test Summary Report # ------------------- # t/01_Bogus.t (Wstat: 0 Tests: 0 Failed: 0) # Parse errors: No plan found in TAP output # Files=1, Tests=0, 1 wallclock secs ( 0.02 usr 0.00 sys + 0.04 cusr 0.00 csys = 0.06 CPU) # Result: FAIL # Failed 1/1 test programs. 0/0 subtests failed. # make[4]: *** [test_dynamic] Error 255 # make[4]: Leaving directory `/tmp/QZ_WiyY7O7' # (make test exited with 512) # CPAN::Reporter: Test result is 'fail', One or more tests failed. # CPAN::Reporter: preparing a CPAN Testers report for Bogus-Module-1.23 # Do you want to edit the test report? (yes/no) [no] no # CPAN::Reporter: sending test report with 'fail' to no_one@example.com, JOHNQP@cpan.org # # # STDERR: # # Failed test 't-NoOutput: 'make test' grade explanation correct' # at t/52_test_report_unknown.t line 69. # got: 'One or more tests failed.' # expected: 'No tests were run.' # Failed test 't-NoOutput: 'perl Build test' returned 1' # at t/52_test_report_unknown.t line 69. # Failed test 't-NoOutput: 'perl Build test' grade reported as 'unknown'' # at t/52_test_report_unknown.t line 69. # got: 'fail' # expected: 'unknown' # STDOUT: # Copying lib/Bogus/Module.pm -> blib/lib/Bogus/Module.pm # t/01_Bogus...... No subtests run # # Test Summary Report # ------------------- # t/01_Bogus.t (Wstat: 0 Tests: 0 Failed: 0) # Parse errors: No plan found in TAP output # Files=1, Tests=0, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.04 cusr 0.01 csys = 0.07 CPU) # Result: FAIL # Failed 1/1 test programs. 0/0 subtests failed. # (/home/src/perl/repoperls/installed-perls/perl/pKhaQOm/perl-5.8.0@32603/bin/perl Build test exited with 65280) # CPAN::Reporter: Test result is 'fail', One or more tests failed. # CPAN::Reporter: preparing a CPAN Testers report for Bogus-Module-1.23 # Do you want to edit the test report? (yes/no) [no] no # CPAN::Reporter: sending test report with 'fail' to no_one@example.com, JOHNQP@cpan.org # # # STDERR: # # Failed test 't-NoOutput: 'perl Build test' grade explanation correct' # at t/52_test_report_unknown.t line 69. # got: 'One or more tests failed.' # expected: 'No tests were run.' # Looks like you failed 6 tests of 46. Dubious, test returned 6 (wstat 1536, 0x600) Failed 6/46 subtests Hope this helps, I have no time left to investigate.

Tue Dec 11 08:37:04 2007 dagolden [...] cpan.org - Correspondence added

From:

DAGOLDEN [...] cpan.org

On Tue Dec 11 03:20:13 2007, ANDK wrote: Show quoted text

> # Failed test 't-NoOutput: 'make test' grade reported as 'unknown'' > # at t/52_test_report_unknown.t line 69. > # got: 'fail' > # expected: 'unknown'

Looks like 3.05 treats no output as a test failure instead of unknown. When I run t/dist/No-Output manually, I get "Parse errors: No plan found in TAP output". If that's the new standard, I can update CPAN::Reporter, but that's not consistent with the old behavior of Test::Harness. David

Tue Dec 11 08:37:08 2007 The RT System itself - Status changed from 'new' to 'open'

Tue Dec 11 09:08:41 2007 andy [...] hexten.net - Correspondence added

On Tue Dec 11 08:37:04 2007, DAGOLDEN wrote: Show quoted text

> On Tue Dec 11 03:20:13 2007, ANDK wrote:

> > # Failed test 't-NoOutput: 'make test' grade reported as 'unknown'' > > # at t/52_test_report_unknown.t line 69. > > # got: 'fail' > > # expected: 'unknown'

> > Looks like 3.05 treats no output as a test failure instead of unknown. > When I run t/dist/No-Output manually, I get "Parse errors: No plan found > in TAP output". > > If that's the new standard, I can update CPAN::Reporter, but that's not > consistent with the old behavior of Test::Harness.

Sorry David, Andreas. Once we resolve this problem I'll add a run of CPAN::Reporter's tests to one of our smoke boxes which should ensure it can't happen again. The current behaviour is return ( $has_errors || $total != $passed ) ? 'FAIL' : $total ? 'PASS' : 'NOTESTS'; $has_errors covers things like 'no plan in TAP'. David: this feature exists primarily for CPAN::Reporter - so it should do what you want it to. Can we bottom out what the semantics should be? Currently: PASS: A non-zero number of tests passed, no parse errors in TAP NOTESTS: No tests run (perhaps all skipped) and no errors in TAP FAIL: Test failures, TAP errors A test that outputs nothing - or produces output that has TAP parse errors is a fail. prove t/dist/t-NoOutput/t/01_Bogus.t t/dist/t-NoOutput/t/01_Bogus...... No subtests run Test Summary Report ------------------- t/dist/t-NoOutput/t/01_Bogus.t (Wstat: 0 Tests: 0 Failed: 0) Parse errors: No plan found in TAP output Files=1, Tests=0, 0 wallclock secs ( 0.01 usr 0.01 sys + 0.02 cusr 0.00 csys = 0.04 CPU) Result: FAIL We need to decide: should a test that outputs nothing be FAIL or NOTESTS. I think it's a FAIL because NOTESTS means 'no tests expected' rather than 'no tests run'.

Tue Dec 11 09:13:42 2007 andy [...] hexten.net - Correspondence added

As an aside I didn't initially see this because it's in the CPAN::Reporter queue. It should probably be in T::H's queue.

Tue Dec 11 09:30:30 2007 dagolden [...] cpan.org - Correspondence added

On Tue Dec 11 09:08:41 2007, ANDYA wrote: Show quoted text

> David: this feature exists primarily for CPAN::Reporter - so it should > do what you want it to. > Can we bottom out what the semantics should be?

I put the question to perl-qa and tapx-dev. You should see those shortly if you haven't already. We can do this either way once everyone agrees on what the right semantics should be. Either way, it looks like I'll have to update CPAN::Reporter. I just added a new t-NoOutput-die test distribution (call Test::More then use a non-existent module) and T::H 2.64 doesn't see that as a failure, but as unknown. So in that sense, T::H 3.05 is right to flag non-zero exit as failure, regardless of output. So that means more fun custom testing logic depending on whether we have 2.XX, 3.0[0-4], 3.05, 3.0[>5] and so on. David

Wed Dec 12 09:02:29 2007 dagolden [...] cpan.org - Correspondence added

Fixed in 1.0601

Wed Dec 12 09:02:34 2007 dagolden [...] cpan.org - Status changed from 'open' to 'resolved'

Bug #31373 for CPAN-Reporter: Test::Harness 3.05 breaks CPAN::Reporter

Preferred bug tracker