Show quoted text> > We are building a test suite with dependencies across the tests, so
> > they must be run in strict order. Because of the large amounts of
> > tests to be included, we have to split them into multiple modules,
> > running the tests of each in order. The test methods within each
> > module run in case-sensitive alphabetical order, but the modules
> > themselves appear to be run in random order.
> >
> > Currently using the Test::Class version 0.41, and the following patch
> > appears to work for us:
> >
> > 365c365
> > < TEST_OBJECT: foreach my $t (@tests) {
> > ---
> > > TEST_OBJECT: foreach my $t (sort @tests) {
> >
> > This change will effectively add a new step ("All of the modules in
> > alphabetical order") before the first step on the documented section
> > "RUNNING ORDER OF METHODS".
>
> Speaking as someone who (a) does not claim to be an expert on
> Test::Class but (b) does use Test::Class on $job ...
>
> I do not think the change requested above should be implemented.
> In most smoke testing setups that I am aware of, you should not want
> "a test suite with dependencies across the tests", as that would
> prevent you from making use of parallelism in testing, e.g.,
> 'make -j8 test'. The fact that the various test classes *can* be
> run in an indeterminate order is, IMO, a feature, not a bug.
>
> If the original posters want to sub-class Test::Class for that
> purpose, then more power to them. But I do not think we should
> change Test::Class's behavior for this purpose.
I don't consider this as a bug per se, but rather a lack of documentation and/or a deviance from the behavior elsewhere in the module. The running order of the methods is clearly defined within the test classes. The test classes can also be run in order if they are provided as parameters to runtests(); which would actually break with my originally suggested patch, but putting the sort around _test_classes() should be safe:
< @tests = _test_classes( $base_class );
---
Show quoted text> @tests = sort _test_classes( $base_class );
Normally we indeed do not want any dependencies between tests, and only ran into this when creating tests for setting up the configuration of our system -- the configuration objects have references to each other, requiring certain objects to be created before others. We have also already worked around this case by creating our own objects establishing the order of tests, and are currently using an unpatched vanilla version of Test::Class.
An alternative fix for this issue would be just to document this behavior, i.e. deterministic order when test classes are given as parameters and nondeterministic otherwise. I will leave it to the maintainers to decide which approach would be more suitable.
