| CC: | Perl Porters <perl5-porters [...] perl.org> |
| Subject: | Counting bug in Test::Harness with unexpected early exit |
| Date: | Fri, 1 Jun 2007 16:20:56 -0400 (EDT) |
| To: | bug-test-harness [...] rt.cpan.org |
| From: | Andy Dougherty <doughera [...] lafayette.edu> |
Test::Harness gives wrong numbers when the test script dies unexpectedly.
This happens for both bleadperl and perl-5.8.8.
Consider the following simple test file bug.t:
use Test;
BEGIN { plan tests => 6 }
ok(1);
ok(0);
ok(0);
ok(0);
die("unexpected crash!");
ok(5);
ok(6);
Running ./perl -Ilib utils/prove bug.t produces the incorrect result:
bug....NOK 2/6# Failed test 2 in bug.t at line 4
# bug.t line 4 is: ok(0);
bug....NOK 3/6# Failed test 3 in bug.t at line 5
# bug.t line 5 is: ok(0);
bug....NOK 4/6# Failed test 4 in bug.t at line 6
# bug.t line 6 is: ok(0);
unexpected crash! at bug.t line 7.
bug....dubious
Test returned status 255 (wstat 65280, 0xff00)
DIED. FAILED tests 2-6
Failed 5/6 tests, 16.67% okay
Failed Test Stat Wstat Total Fail List of Failed
-------------------------------------------------------------------------------
bug.t 255 65280 6 7 2-6
Failed 1/1 test scripts. 5/6 subtests failed.
Files=1, Tests=6, 0 wallclock secs ( 0.01 cusr + 0.00 csys = 0.01 CPU)
Failed 1/1 test programs. 5/6 subtests failed.
Note in the table that it claims there are 6 tests (correct) but that 7 failed.
However, in the summary, it correctly notes that 5 out of the 6 tests
failed.
perl-5.8.8 fails in a similar way, but with the added incorrect
percentage of 116.67% failing.
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
bug.t 255 65280 6 7 116.67% 2-6
I speculate that it's double-counting all tests after the die().
For an extreme example, consider the output I get from running 'make
test' for parrot:
Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/compilers/pge/p5regex/p5rx.t 0 3 960 1466 152.71% 228-960
t/compilers/pge/p6regex/01-regex. 0 3 ?? ?? % ??
t/examples/shootout.t 1 256 20 1 5.00% 16
t/pmc/complex.t 5 1280 53 5 9.43% 32 42-43 46-47
t/src/hash.t 1 256 11 1 9.09% 6
t/src/io.t 255 65280 20 18 90.00% 12-20
(4 subtests UNEXPECTEDLY SUCCEEDED), 11 tests and 277 subtests skipped.
Failed 6/304 test scripts, 98.03% okay. 449/6669 subtests failed, 93.27% okay.
where the numbers listed as faling for p5rx.t and io.t are exactly twice the number
you get from the "List of Failed". Note too that there is no obvious
way to get the number '449' in the summary from any of the numbers in
the table above!
--
Andy Dougherty doughera@lafayette.edu