| Subject: | Plan / Tests Mismatch |
| Date: | Thu, 08 Jul 2010 12:48:15 -0500 |
| To: | bug-TAP-Formatter-JUnit [...] rt.cpan.org |
| From: | Phillip Kimmey <pkimmey [...] genome.wustl.edu> |
I am also using TAP::Formatter::JUnit for Hudson CI testing, and found
Hudson wasn't reporting mismatches between number of planned tests and
actual tests run.
Although the <testsuite> tag will an errors="#" that represents this
mismatch, an actual <failure> tag is needed to trigger Hudson's error
reporting.
Colin Robertson (ticket #58838) contributed a fix that I believe should
catch build fails, but doesn't catch these plan / run mismatches.
I've attached my diff, which integrates his changes to
Tap/Formatter/JUnit/Session.pm, but does not include his additional test
cases.
I'm not sure whether this mismatch should be considered a "fail" or an
"error," but I set it as a "fail" for now.
Please let me know if there's anything else I can do to help,
Thanks,
Phil Kimmey
--- Session.pm.back 2010-07-06 17:37:08.000000000 -0500
+++ Session.pm 2010-07-08 11:15:17.000000000 -0500
@@ -77,12 +77,14 @@
$self->_flush_queue;
# if the test died unexpectedly, make note of that
+ my $die_msg;
my $exit = $parser->exit();
if ($exit) {
my $sys_err = $self->system_err;
my $wstat = $self->parser->wait();
my $status = sprintf( "%d (wstat %d, 0x%x)", $exit, $wstat, $wstat );
- $sys_err .= "Dubious, test returned $status\n";
+ $die_msg = "Dubious, test returned $status";
+ $sys_err .= $die_msg . "\n";
$self->system_err($sys_err);
}
@@ -117,8 +119,21 @@
$errors += $parser->todo_passed() unless $self->passing_todo_ok();
$errors += abs($testsrun - $planned) if ($planned);
$errors += ($noplan || $bad_exit);
+
+ if ($planned and $testsrun != $planned) {
+ my %attrs = (
+ 'name' => "Test plan validity",
+ ($self->formatter->timer ? ('time'=>$self->_time_since_last_test) : ()),
+ );
+ my $failure = $xml->failure({
+ type => "PlanMismatch",
+ message => " Number of planned tests does not match with run tests: expected " . $planned . " but ran " . $testsrun . " tests.",
+ });
+ my $testcase = $xml->testcase(\%attrs, $failure);
+ $self->add_testcase($testcase);
+
+ }
- my @tests = @{$self->testcases()};
my %attrs = (
'name' => _get_testsuite_name($self),
'tests' => $testsrun,
@@ -126,7 +141,20 @@
'failures' => $failures,
'errors' => $errors,
);
- my $testsuite = $xml->testsuite(\%attrs, @tests, $sys_out, $sys_err);
+
+ my $error;
+ if ($die_msg) {
+ $error = $xml->error({ message => $die_msg });
+ }
+ elsif($noplan) {
+ $error = $xml->error({ message => "No plan in TAP output" });
+ }
+
+ my @tests = @{$self->testcases()};
+ my @elements = (@tests, $sys_out, $sys_err);
+ unshift @elements, $error if $error;
+
+ my $testsuite = $xml->testsuite(\%attrs, @elements);
$self->formatter->add_testsuite($testsuite);
$self->dump_junit_xml($testsuite);
}