| Subject: | Randomizing tests |
Add randomization of test order to Test::Harness.
See attached file for some extended thoughts...
OK, you said this would be problematic, but I figured I should
add it to rt anyway. (Plus, I wanted to play with rt.)
So, I was going to just put here 'randomize order by which test
scripts are invoked in runtests()'. But I started thinking (and this
started getting long), and I'm wondering, regardless of practicality,
if randomizing on the file order really makes sense.
I've always been under the impression that each test script should
undo any changes it's made. E.g., each script creates an object it
needs to make, and destroys it by the time it's finished, rather than
01new.t creating an object, and 02set.t testing mutators on the object
created by 01new.t. Or that you shouldn't have 01write.t create a
file and 02read.t read from that same file. (Whether this is correct,
you might want to touch on this briefly in your presentation, when you
get to the logic of writing multiple scripts.)
If my understanding of this isn't correct, and it's acceptable or
desirable for each test script to depend on actions taken by previous
test scripts, then the whole notion of randomizing the run order of
the test scripts is a non-starter, at least for any situation where
that sort of ordering is required.
If my understanding is correct, and test scripts should leave no trace
when they finish, then randomization at the level of scripts is
pointless, since each test script will start out with essentially a
tabula rasa. (Unless the test scripts themselves are broken and fail
to undo their changes, or if they crash midway through, before getting
the chance to undo changes.)
So, what's needed is randomization at the level of the individual
tests, within a single script. I've got no idea how to do this using
the current situation where tests are just simple Perl scripts. (One
way would be to read the file, and juggle all lines that execute
tests, and eval the result; but this is obviously a dumb idea.)
There's probably no way to do this at all so that you could just take
a standard test script and arbitrarily toggle between random and
sequential testing.
You'd have to require the script writer to set up the script to be
random from the get-go. Basically, the script writer would take each
test to be randomized (along with any ancillary set-up or tear-down
code that the test depends upon) and wrap the code into a variable.
(I.e. as a sub, or a code reference, or as a string, or maybe as a
labelled block, or possibly even within pod directives.) Then, at the
end of the script, a randomization function would be invoked with the
names/code references/labels/whatever of each test to be randomized.
Tests in the body of the script that aren't wrapped up and set aside
for randomization would be executed first, in sequence, as with any
normal test script. The randomization function could first look at
the script's input to see if a record of a previous run was passed in
(either via STDIN or as a file to open) and if there is, replicate
that order; otherwise by default it would generate a new random order.
So, basically, we'd have to have a Test::Randomize module that would
do a lot of this work, except for the bit where the test writer has to
chop up the script into randomizable chunks and pass them into a
run_randomly() function.
But is this even something that's really needed? Is it worth the
extra work?