Skip Menu |

This queue is for tickets about the Test-Most CPAN distribution.

Report information
The Basics
Id: 104465
Status: open
Priority: 0/
Queue: Test-Most
People
Owner: Nobody in particular
Requestors: sebastian_rose [...] gmx.de
Cc:
AdminCc:
Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)

History Show all quoted text
  Fri May 15 16:37:34 2015 sebastian_rose [...] gmx.de - Ticket created
Subject: Severe Bug: tests run after bail_on_fail
Date: Fri, 15 May 2015 22:37:17 +0200
To: bug-Test-Most [...] rt.cpan.org
From: Sebastian Rose <sebastian_rose [...] gmx.de>
Hello maintainers, I just (2015-05-15) installed Test::Most. I was especially interested in the handy error handling features. Unfortunately I found, what I consider a severe bug (on the first glance). A few examples:
use Test::Most 'no_plan'; die_on_fail; ok 1, 'Throw on fail'; restore_fail; # The error happens with and without this line. bail_on_fail; ok 0, 'BAIL_OUT on fail'; # Expected to stop testing immediately! restore_fail; ok 1, 'Dangerous test';
sh# prove t/test_most_error_handling.t -v 01 t/test_most_error_handling.t .. 02 ok 1 - Throw on fail 03 not ok 2 - BAIL_OUT on fail 04 ok 3 - Dangerous test - WILL BE EXECUTED 05 1..3 06 Bailout called. Further testing stopped: Test failed. BAIL OUT!. 07 Bail out! Test failed. BAIL OUT!. 08 09 # Failed test 'BAIL_OUT on fail' 10 # at t/test_most_error_handling.t line 9. 11 # Looks like you failed 1 test of 3. 12 FAILED--Further testing stopped: Test failed. BAIL OUT!. NOTE lines 4 and 11! 3 Tests were executed - instead of 2. According to the docs, this does not seem to be the intended behaviour (to me). Here comes the behaviour that I would expect (triggering BAIL_OUT "by hand"):
use Test::Most 'no_plan'; die_on_fail; ok 1, 'Throw on fail'; bail_on_fail; ok( 0, 'BAIL_OUT on fail' ) or BAIL_OUT(" S T O P "); restore_fail; ok 1, 'Dangerous test - WILL NOT BE EXECUTED';
sh# prove t/test_most_error_handling.t -v 01 t/test_most_error_handling.t .. 02 ok 1 - Throw on fail 03 not ok 2 - BAIL_OUT on fail 04 Bailout called. Further testing stopped: S T O P 05 Bail out! S T O P 06 07 # Failed test 'BAIL_OUT on fail' 08 # at t/test_most_error_handling.t line 7. 09 FAILED--Further testing stopped: S T O P Only two tests executed. BAIL_OUT stopped the testing immediatly. 'Dangerous test' is NOT executed. -- Ich setzte einen Fuß in die Luft, und sie trug. (Hilde Domin)
  Fri May 15 17:08:43 2015 ether [...] cpan.org - Correspondence added
On 2015-05-15 13:37:34, sebastian_rose@gmx.de wrote: Show quoted text
> Hello maintainers, > > > I just (2015-05-15) installed Test::Most. I was especially interested > in the handy error handling features. > > Unfortunately I found, what I consider a severe bug (on the first > glance). A few examples: > >
What is your installed version of Test::More and Test::Builder?
  Fri May 15 17:08:44 2015 The RT System itself - Status changed from 'new' to 'open'
  Fri May 15 17:17:24 2015 exodist7 [...] gmail.com - Correspondence added
I seem to recall a discussion about this, and a Test::Builder feature request that never got fleshed out. Basically if you bail-out as soon as the failure happens you get no debugging, you simply get the 'not ok' line and no info on what line it failed. So the way this feature is often implemented is to note that you should bail, but let stuff continue so that diag gets printed, then bail the next time ok is called. So you have a choice between immediate bailing with no helpful diag, or helpful diag, but a delayed bail. Here are related discussions, I may have some details wrong, so if the conversation contradicts anything above believe the conversation: https://github.com/Test-More/test-more/issues/373 https://github.com/Test-More/test-more/wiki/Stop-On-Fail
  Fri May 15 17:43:21 2015 sebastian_rose [...] gmx.de - Correspondence added
Subject: Re: [rt.cpan.org #104465] Severe Bug: tests run after bail_on_fail
Date: Fri, 15 May 2015 23:43:06 +0200
To: "Karen Etheridge via RT" <bug-Test-Most [...] rt.cpan.org>
From: Sebastian Rose <sebastian_rose [...] gmx.de>
"Karen Etheridge via RT" <bug-Test-Most@rt.cpan.org> writes: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=104465 > > > On 2015-05-15 13:37:34, sebastian_rose@gmx.de wrote:
>> Hello maintainers, >> >> >> I just (2015-05-15) installed Test::Most. I was especially interested >> in the handy error handling features. >> >> Unfortunately I found, what I consider a severe bug (on the first >> glance). A few examples: >> >>
> > > What is your installed version of Test::More and Test::Builder? >
Hm - sorry - the info was missing. I updated all my Perl modules just before I tried Test::Most. $Test::More::VERSION = 1.001014 $Test::Builder::VERSION = 1.001014 This is perl 5, version 20, subversion 1 (v5.20.1) built for x86_64-linux-thread-multi -- Ich setzte einen Fuß in die Luft, und sie trug. (Hilde Domin)
  Fri May 15 17:49:32 2015 sebastian_rose [...] gmx.de - Correspondence added
Subject: Re: [rt.cpan.org #104465] Severe Bug: tests run after bail_on_fail
Date: Fri, 15 May 2015 23:49:21 +0200
To: "Chad Granum via RT" <bug-Test-Most [...] rt.cpan.org>
From: Sebastian Rose <sebastian_rose [...] gmx.de>
"Chad Granum via RT" <bug-Test-Most@rt.cpan.org> writes: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=104465 > > > I seem to recall a discussion about this, and a Test::Builder feature request > that never got fleshed out. Basically if you bail-out as soon as the failure > happens you get no debugging, you simply get the 'not ok' line and no info on > what line it failed. So the way this feature is often implemented is to note > that you should bail, but let stuff continue so that diag gets printed, then > bail the next time ok is called. > > So you have a choice between immediate bailing with no helpful diag, or helpful diag, but a delayed bail. > > Here are related discussions, I may have some details wrong, so if the conversation contradicts anything above believe the conversation: > > https://github.com/Test-More/test-more/issues/373 > https://github.com/Test-More/test-more/wiki/Stop-On-Fail
Hallo Chad, the problem is, that the docs say BAIL_OUT is called. I would expect the tests to stop. "bail_on_fail" bail_on_fail; is_deeply $foo, bar, '... we bail out if this fails'; This function, if called, will cause the test suite to BAIL_OUT() if any tests fail after it. ... If the "DIE_ON_FAIL" or "BAIL_ON_FAIL" environment variables are true, any tests which use "Test::Most" will throw an exception or call BAIL_OUT on test failure. perldoc Test::More BAIL_OUT BAIL_OUT($reason); Indicates to the harness that things are going so badly all testing should terminate. This includes the running of any additional test scripts. This is typically used when testing cannot continue such as a critical module failing to compile or a necessary external utility not being available such as a database connection failing. The test will exit with 255. For even better control look at Test::Most. Debugging can be done differently (and why not add more tests to find an error?). -- Ich setzte einen Fuß in die Luft, und sie trug.
  Fri May 15 21:08:29 2015 exodist7 [...] gmail.com - Correspondence added
Test::Most can document/promise whatever it wants, that doesn't change the technical issues it would have to overcome to do what it promises while still using Test::Builder under the hood. I do not maintain Test::Most (thank God), but whoever does needs to see how Test::Most implements die on fail and either fix it as a bug, or document its behavior better. I just wanted to make sure everyone involved knew about the Test::Builder architectural issues that had bearing on this bug.