Skip Menu |

This queue is for tickets about the Perl-Dist-Strawberry CPAN distribution.

Report information
The Basics
Id: 58545
Status: open
Priority: 0/
Queue: Perl-Dist-Strawberry
People
Owner: Nobody in particular
Requestors: adamk [...] cpan.org
Cc:
AdminCc:
Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)

History Show all quoted text
  Sun Jun 20 22:41:01 2010 adamk [...] cpan.org - Ticket created
Subject: Strawberry needs a post-build QA test process
Over the last 6 months or so, most of the serious bugs that have been reported relate to mistakes that we really should have picked up in beta testing, but didn't. Examples: Bad PPM repository, badly broken C stuff, distributions missing support DLLs etc. I think we've seen enough of these that we should consider this to be a systematic flaw in the Strawberry production process rather than a collection of small oversights. I'd like to propose we develop a post-installation QA mechanism, so that beta testers and other can drop the installer onto a machine, fire up CPAN and just "install Perl::Dist::Strawberry::QA" or something similar. These post-production tests would potentially require that the test machines have MySQL, Postgres etc databases installed on them, and have Test::Database configured. It might try some basic tests for MySQL and Postgres (not necesarily the entire test suite), SQLite, installation via PPM, etc (installing the QA module itself tests CPAN installation). We could add a series of different regression tests to this module as we uncover and fix various problems which aren't caught by the module's own test scripts.
  Mon Jun 21 01:45:16 2010 csjewell [...] cpan.org - Correspondence added
On Sun Jun 20 22:41:01 2010, ADAMK wrote: Show quoted text
> Over the last 6 months or so, most of the serious bugs that have been > reported relate to mistakes that we really should have picked up in beta > testing, but didn't. > > Examples: Bad PPM repository, badly broken C stuff, distributions > missing support DLLs etc. > > I think we've seen enough of these that we should consider this to be a > systematic flaw in the Strawberry production process rather than a > collection of small oversights. > > I'd like to propose we develop a post-installation QA mechanism, so that > beta testers and other can drop the installer onto a machine, fire up > CPAN and just "install Perl::Dist::Strawberry::QA" or something similar. > > These post-production tests would potentially require that the test > machines have MySQL, Postgres etc databases installed on them, and have > Test::Database configured. > > It might try some basic tests for MySQL and Postgres (not necesarily the > entire test suite), SQLite, installation via PPM, etc (installing the QA > module itself tests CPAN installation). > > We could add a series of different regression tests to this module as we > uncover and fix various problems which aren't caught by the module's own > test scripts.
Personally, I agree. I think instead of a CPAN dist, we need to make it a script that we can wrap with PAR::Packer that would run on a clean system. (Because we can't really QA some things well with a working Strawberry already installed, is my thought.) Fun first problem (and this would catch a few different types of problems): A post-build check for missing files between the .zip and .msi. It's easy enough to code... you just could not start with Strawberry already installed. (We still need to be able to QA-test 5.10, which still has hard-coding.) Strawberry would have to be extracted from the .zip, a File::List::Object object created, and then deleted. Then we'd install Strawberry from the .msi into the same directory, grab another FLO object created, and uninstall, and then we'd do along the lines of this script: http://cpansearch.perl.org/src/CSJEWELL/File-List-Object-0.202/eg/filecheck.pl Maybe fail if the count is not 0 for either one, or something. That'd be test 1.
  Mon Jun 21 01:45:19 2010 The RT System itself - Status changed from 'new' to 'open'