Bug #85137 for Module-CPANTS-Analyse: New kwalitee metric: look for shebangs in tests

Wed May 08 12:13:38 2013 ether [...] cpan.org - Ticket created

Subject:

New kwalitee metric: look for shebangs in tests

Somewhat related to RT#84921. Files under t/ (normally run by the test harness - make test or ./Build.PL test) should not have any shebang lines in them at all - they are not invoked as executables, and should always be tested with the perl being used to install the distribution.

Wed May 08 12:14:05 2013 ether [...] cpan.org - Severity Wishlist added

Wed May 08 12:14:30 2013 ether [...] cpan.org - Reference to ticket #84921 added

Wed May 08 17:42:46 2013 daxim [...] cpan.org - Correspondence added

That isn't right, test shebangs sets taint warnings or checks. There's special handling for that in TAP::Parser::SourceHandler::Perl. Does this make this bug invalid? If not, redefine/add more info.

Wed May 08 17:42:47 2013 The RT System itself - Status changed from 'new' to 'open'

Wed May 08 17:49:50 2013 ether [...] cpan.org - Correspondence added

On 2013-05-08 14:42:46, DAXIM wrote: Show quoted text

> That isn't right, test shebangs sets taint warnings or checks. There's > special handling for that in TAP::Parser::SourceHandler::Perl. Does > this make this bug invalid? If not, redefine/add more info.

It's been a long time since I've used taint, so I'll have to look into the different ways to set this... but in the meantime, I can rephrase as "don't fail this test if the shebang has a -T". :)

Thu May 09 01:41:46 2013 ishigaki [...] cpan.org - Correspondence added

Besides the -T issue, have shebangs in tests caused actual problems for toolchains (or anywhere)? This seems to me something for Perl::Critic. On Thu May 09 01:13:38 2013, ETHER wrote: Show quoted text

> Somewhat related to RT#84921. > > Files under t/ (normally run by the test harness - make test or > ./Build.PL test) should not have any shebang lines in them at all - > they are not invoked as executables, and should always be tested > with the perl being used to install the distribution.

Thu May 09 12:19:46 2013 ether [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #85137] New kwalitee metric: look for shebangs in tests
Date:	Thu, 9 May 2013 09:19:21 -0700
To:	Kenichi Ishigaki via RT <bug-Module-CPANTS-Analyse [...] rt.cpan.org>
From:	Karen Etheridge <ether [...] cpan.org>

On Thu, May 09, 2013 at 01:41:47AM -0400, Kenichi Ishigaki via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=85137 > > > Besides the -T issue, have shebangs in tests caused actual problems for toolchains (or anywhere)? This seems to me something for Perl::Critic.

Yes, this came up yesterday regarding Daemon::Control being packaged up into an rpm -- the wrong perl was being invoked. But you bring up a point which I don't think (to my knowledge at least) has been clearly articulated anywhere -- what things are appropriate as kwalitee metrics, and what things are better suited as perlcritic metrics? It would be good to have a clear (written) policy, to help guide future metric authors.

Fri May 10 02:19:31 2013 ishigaki [...] cpan.org - Correspondence added

On Fri May 10 01:19:46 2013, ETHER wrote: Show quoted text

> On Thu, May 09, 2013 at 01:41:47AM -0400, Kenichi Ishigaki via RT > wrote:

> > <URL: https://rt.cpan.org/Ticket/Display.html?id=85137 > > > > > Besides the -T issue, have shebangs in tests caused actual problems

> for toolchains (or anywhere)? This seems to me something for > Perl::Critic. > > Yes, this came up yesterday regarding Daemon::Control being packaged > up > into an rpm -- the wrong perl was being invoked.

Thanks, got the point. Though looking for shebangs in all the test files might be somewhat overkill (at least the target should be .pl files in t/ directory for now), we'll eventually need to consider about post-installation tests (see Lancaster consensus), and this "shebangs in test files" information would give us a nice topic for later discussion. I think it's too early to make this a solid Kwalitee metric (cf. #84921), but I'd like to add a code to analyse test files to provide some statistics on the website. Show quoted text

> But you bring up a point which I don't think (to my knowledge at > least) has > been clearly articulated anywhere -- what things are appropriate as > kwalitee metrics, and what things are better suited as perlcritic > metrics? > It would be good to have a clear (written) policy, to help guide > future > metric authors.

Check first: http://qa.perl.org/phalanx/kwalitee.html For me, (1) metrics not to break toolchains, or to alleviate a burdon on toolchain maintainers (eg. extraction, meta validation) -> Kwalitee (2) metrics for security or for portability (eg. symlinks) -> Kwalitee (3) metrics every CPAN author should test by themselves but often forget/ignore (eg. pod) -> Kwalitee (4) metrics generally useful but controversial on occasion (eg. coding style, module selection, other best practices irrelevant to toolchains) -> Critic (which is customizable) (5) bugs specific to a dist/an author/a group -> bug trackers (6) metrics to gather information from the whole CPAN/BackPAN -> depends, but probably CPANTS can help as it actually extracts every distribution (7) others (eg. popularity/repackageability) -> depends, but maybe they should be done somewhere else for their specific audience. Is there any other aspect that should be covered?