| Subject: | FEATURE SPEC - Overloadable straps class |
====================================================================
Specification - Proposed Test::Harness Feature - Overloadable Straps
====================================================================
1. Problem Scope
----------------
While Test::Harness provides a standard method by which to run tests,
in some situations a much greater granularity of testing is desirable.
Because test scripts are run as normal programs, and output results
in a standard format (TAP) it becomes theoretically possible for the
entire output of a test run to be captured, and subsequently "replayed".
This is particularly attractive for large scale testing, as it means
that the testing results can be captured and stored with maximum
granularity, and the actual analysis of the test results can be
undertaken seperately, subsequent to the actual test run.
This seperation of concens has other benefits. For example, if a bug
is found in the Test::Harness code, a previous test run could
concievable be replayed back through the new code, yielding new and
more accurate testing results for the past test run.
However, these large scale testing systems will also trend towards
maximum accuracy and compatibility with the mechanisms by which the
tests are run.
That is, a testing system would ideally NOT be able to interact with
Test::Harness at a Perl code level, as it will limit itself to merely
executing the build sequence (e.g. perl Makefile.PL; make; make test)
at a shell level, rather than substitute an entire alternate testing
environment.
Such systems would also desire to do the capture of the raw output
as close to the scripts as possible, with as little code sitting
between the capture logic and the test scripts as is reasonable.
2. Implementation Concept
-------------------------
In prototypes of these types of automated testing, it has been found
that the ONLY way of reliable interacting with the testing environment
that can be replicated and used across a wide range of languages,
testing schemes and plans is by the use of environment variables.
Such systems can also create and provide additional code modules,
although they have no way of running them.
Thus, from the point of view of Test::Harness, an implementation of
a system for allowing an external party to capture the test output
would consist of the following.
a) An environment variable is provided by the testing system
b) Test::Harness checks for this environment variable being set
c) Test::Harness loads the module provided by the external party
d) Test::Harness continues with the test process as normal
e) Code in the external module captures the output and dispatches
it in whatever manner is desired by the testing system.
3. Implementation Specifics
---------------------------
Before launching "make test" or equivalent, the test system will define
the environment variable HARNESS_STRAP_CLASS.
The variable can contain one of two values. Either the name of a class
that exists within the @INC path for the perl being used (if the value
does NOT end in ".pm") or the full path to a Perl module to be loaded
by the string form of require, and which returns as it's "true value"
the name of the class that it contains.
Before the creation of the first test straps object, Test::Harness will
check the HARNESS_STRAP_CLASS environment variable.
If a class name, it will attempt to load the class, otherwise for a
file name it will load the file, and verify that the class name returned
exists and ->isa('Test::Harness::Straps'), or otherwise will use the
default test straps class as normal.
Instead of using hard-coded Test::Harness::Straps, Test::Harness will
instead use the class name determined early, which should implement the
exact duplicate API as the default class, most likely by the use of
the Decorator or Adapter pattern (although any such implementation falls
outside the scope of this specification).
Test::Harness will then continue as normal.
As long as it follows the same API as for the default class, the custom
class shall undertake any additional tasks or implement any additional
functionality as is needed to implement the custom functionality.