Bug #61462 for Test-Simple: Reports of Test-Simple-2.00

Sun Sep 19 13:40:40 2010 ANDK [...] cpan.org - Ticket created

CC:	JMCNAMARA [...] cpan.org,MIYAGAWA [...] cpan.org
Subject:	Test-Simple 2.00_01 breaks Twiggy-0.1007 and Spreadsheet-ParseExcel-0.58

As per subject, I just found on my smokers two victims (John and Miyagawa) of Test-Simple 2.00_01. Here are the analysis pages pointing it out: http://analysis.cpantesters.org/solved?distv=Twiggy-0.1007 http://analysis.cpantesters.org/solved?distv=Spreadsheet-ParseExcel-0.58 I apologize to the aggrieved parties and hope they are not too frustrated by the red flags on their testing matrix. I've now uninstalled Test::Simple 2.00_01 from all my smokers to prevent further frustration among more authors. Thanks and best wishes to everybody involved!

Sun Sep 19 17:08:40 2010 schwern [...] pobox.com - Correspondence added

Subject:	Re: [rt.cpan.org #61462] Test-Simple 2.00_01 breaks Twiggy-0.1007 and Spreadsheet-ParseExcel-0.58
Date:	Sun, 19 Sep 2010 14:08:25 -0700
To:	bug-Test-Simple [...] rt.cpan.org
From:	Michael G Schwern <schwern [...] pobox.com>

On 2010.9.19 10:40 AM, Andreas Koenig via RT wrote: Show quoted text

> As per subject, > > I just found on my smokers two victims (John and Miyagawa) of > Test-Simple 2.00_01. Here are the analysis pages pointing it out: > > http://analysis.cpantesters.org/solved?distv=Twiggy-0.1007 > http://analysis.cpantesters.org/solved?distv=Spreadsheet-ParseExcel-0.58 > > I apologize to the aggrieved parties and hope they are not too > frustrated by the red flags on their testing matrix. > > I've now uninstalled Test::Simple 2.00_01 from all my smokers to prevent > further frustration among more authors. > > Thanks and best wishes to everybody involved!

Thanks for trying it out, this is really valuable information! Please continue to report these failures to us. I'll look into it more in a bit. Looks like in Twiggy, Test::Builder->use_numbers may be busted or Plack's test suite is doing some schennanigans. Spreadsheet::ParseExcel looks like Test::Builder is failing to detect a $TODO. -- But there's no sense crying over every mistake. You just keep on trying till you run out of cake. -- Jonathan Coulton, "Still Alive"

Sun Sep 19 17:08:41 2010 The RT System itself - Status changed from 'new' to 'open'

Thu Sep 30 15:10:33 2010 ANDK [...] cpan.org - Correspondence added

RT-Send-CC:

DOMQ [...] cpan.org

DOMQ/Test-Group-0.19_01.tar.gz is another example for tests failing only when Test-Simple 2.00_01 is installed. Version/Fail overview: http://analysis.cpantesters.org/reports_by_field?SUBMIT_xxx=Submit&distv=Test-Group-0.19_01&field=mod%3ATest%3A%3ABuilder&field=meta%3Aperl&field=mod%3ATest%3A%3ABuilder Example test output: http://www.cpantesters.org/cpan/report/8746119 Seems to be some case intolerance for how to write "skip". Cheers,

Sat Oct 02 23:01:02 2010 ANDK [...] cpan.org - Correspondence added

RT-Send-CC:

NODINE [...] cpan.org

NODINE/Slay-Makefile-Gress-0.07.tar.gz is also a affected.

Sun Oct 24 06:48:31 2010 ANDK [...] cpan.org - Correspondence added

RT-Send-CC:

ZAG [...] cpan.org

ZAG/WebDAO-2.0_1.tar.gz is also affected.

Thu Nov 18 00:36:26 2010 ANDK [...] cpan.org - Correspondence added

Same for NUFFIN/Log-Dispatch-Config-TestLog-0.02.tar.gz

Thu Nov 18 00:47:30 2010 ANDK [...] cpan.org - Correspondence added

same for SHARYANTO/Data-ModeMerge-0.25.tar.gz

Sun Dec 19 07:08:40 2010 ANDK [...] cpan.org - Correspondence added

Same for LEOCHARRE/WordPress-Base-MediaObject-1.02.tar.gz I think it's time to either remove this Test-Simple 2.00_01 release from CPAN or follow up with a bugfix release. Problem is that with so many bugs reported it is unlikely that more bugs are discovered easily on it. And on the other hand the test results for innocent CPAN authors are disturbed. I would understand that they are annoyed. What do you think?

Mon Dec 20 23:31:33 2010 schwern [...] pobox.com - Correspondence added

Subject:	Re: [rt.cpan.org #61462] Test-Simple 2.00_01 breaks Twiggy-0.1007 and Spreadsheet-ParseExcel-0.58
Date:	Tue, 21 Dec 2010 15:31:33 +1100
To:	bug-Test-Simple [...] rt.cpan.org
From:	Michael G Schwern <schwern [...] pobox.com>

On 2010.12.19 11:08 PM, Andreas Koenig via RT wrote: Show quoted text

> Same for LEOCHARRE/WordPress-Base-MediaObject-1.02.tar.gz > > I think it's time to either remove this Test-Simple 2.00_01 release from > CPAN or follow up with a bugfix release. > > Problem is that with so many bugs reported it is unlikely that more bugs > are discovered easily on it. And on the other hand the test results for > innocent CPAN authors are disturbed. I would understand that they are > annoyed. > > What do you think?

I do agree that 2.00_01 is getting a bit long in the tooth. However, it's still giving good data, primarily about holes in the test suite, so it will stand. For example, the WordPress-Base-MediaObject failure reveals that "use Test::Simple 'no_plan'" is broken and was never tested. SpreadSheet-ParseExcel shows escaping # in test names is broken and not tested, etc... Keep spooling up the information in this ticket and they'll be analyzed and turned into actionable bugs once I (or a volunteer) gets around to it. A bugfix release is not so simple as Test::Builder2 is still being rearchitected. The design is now substantially different from alpha 1. Fortunately, the last bits are coming together so alpha 2 is not far off. As for perturbing the test results of CPAN authors, the whole point of alpha releases and an automated, distributed testing system is to find failures before real users do. Keeping a pretty green stripe comes below that. A solution is to fix the matrix display to downgrade or exclude reports which contain alpha dependencies. -- Robrt: People can't win Schwern: No, but they can riot after the game.

Mon Dec 20 23:32:06 2010 mschwern [...] cpan.org - Subject changed from 'Test-Simple 2.00_01 breaks Twiggy-0.1007 and Spreadsheet-ParseExcel-0.58' to 'Reports of Test-Simple-2.00_01 breakages'

Sun Dec 26 15:14:28 2010 ANDK [...] cpan.org - Correspondence added

Spooling:) LEOCHARRE/File-Filename-1.05.tar.gz

Sat Jan 08 04:14:24 2011 mschwern [...] cpan.org - Correspondence added

I've analyzed all the reports and assigned them individual tickets either on Githb <http://github.com/schwern/test-more/issues> or, if the problem is with the distribution, in that distribution's queue. I've noted which ones I plan on fixing before alpha 2 with "blocker". The rest will either hold up alpha 2 too long or are Somebody Else's Problem. Alpha 2 is coming out in the next couple days. I'm going to close up this ticket to make book keeping easier. Please open a new ticket for alpha 2 bugs. Ideally, report them at <https://github.com/schwern/test-more/issues/> and use individual tickets for each module. If this is inconvenient for you I don't mind if you continue to use RT. Thanks for all the reports! Here's the triage and issue links. Test::SharedFork [github 77] does unspeakable things to Test::Builder author tried to fix it for TB2, TB2 changed Twiggy-0.1007 depends Spreadsheet::ParseExcel [github 78] (blocker) bug in TAP escaping Test::Group [github 96] broken because subtests are broken NODINE/Slay-Makefile-Gress-0.07.tar.gz [rt.cpan.org 64600] Testing for specific output Tests should be rewritten Test::Class 0.36 [github 97] (blocker) WebDAO-2.0_1.tar.gz depends NUFFIN/Log-Dispatch-Config-TestLog-0.02.tar.gz [rt.cpan.org 64601] Overrides Test::Builder methods Should be reimplemented as an event watcher Test::Aggregate [github 98] SHARYANTO/Data-ModeMerge-0.25.tar.gz depends LEOCHARRE/WordPress-Base-MediaObject-1.02.tar.gz [github 91] (blocker) use Test::Simple 'no_plan' is broken Also breaks LEOCHARRE/File-Filename-1.05.tar.gz

Sat Jan 08 04:14:25 2011 mschwern [...] cpan.org - Status changed from 'open' to 'resolved'

Bug #61462 for Test-Simple: Reports of Test-Simple-2.00_01 breakages

Preferred bug tracker