Bug #84605 for App-ArchiveDevelCover: Request inclusion of branch and condition coverage in archive reports

Sat Apr 13 05:47:04 2013 jkeenan [...] cpan.org - Ticket created

CC:	paul [...] pjcj.net
Subject:	Request inclusion of branch and condition coverage in archive reports

Domm, Sitting here at the 2013 Perl QA Hackathon in Lancaster. Sitting across from pjcj. Talking about how to improve cpancover.pm and how to obtain and report coverage of the Perl 5 core distribution. Which led me to (finally!) install App-ArchiveDevelCover, about which I hears you speak at the DC-Baltimore Perl Workshop last April (and where I'm speaking next weekend). So I gave archive_devel_cover.pl a whirl. Two points: 1. It would be good to include in the documentation a reminder that you must do 'cover -delete' between runs of Devel::Cover and archive_devel_cover.pl. Otherwise, the latter program says, "This coverage report has already been archived." 2. More importantly, archive_devel_cover.pl appears to be hard-wired to report only statement, sub and total percentages. It omits branch and condition coverage -- which I rely heavily on -- as well as Pod and time coverage -- which I personally don't use. It would really be good to allow the user the same flexibility with respect to the types of coverage archived that she has when using Devel::Cover itself. If, however, that is not feasible, then the archive_devel_cover.pl should report all the types of coverage that the user requested when running Devel::Cover over the code base. See, for example: http://thenceforward.net/Data-Dumper/archive/, where I tried out archive_devel_cover.pl for the very first time. Why would this be valuable? Two reasons: a. When I'm trying to boost the coverage of a module, the last, most difficult steps are the tests I write to boost branch and condition coverage. Right now, when I do a new Devel::Cover run, I have to look at my web site to see the totals for the *previous* cover run, then refresh the browser to see the improvements from the *latest* run. In the later stages of a coverage-improvement project, I've often got all the statement coverage I can possibly get on a given OS, so what I'm looking for are incremental improvements in branch and condition coverage. b. Here at the hackathon we are discussing how to report test coverage of the Perl 5 core distribution on an ongoing basis. That would include 'gcov' running on the C code and Devel::Cover running on all the Perl modules in cpan/, dist/, ext/, lib/, etc. We'll certainly be able to get branch and condition coverage on all the .pm files using Devel::Cover, but it would be very useful as well to be able to track changes in those coverage categories between monthly releases of, say, Perl 5.19. So, can you give us an idea as to whether archiving of branch and condition coverage is feasible and might be forthcoming? Thank you very much. Jim Keenan

Sat Apr 13 08:27:18 2013 domm [...] plix.at - Correspondence added

Subject:	Re: [rt.cpan.org #84605] Request inclusion of branch and condition coverage in archive reports
Date:	Sat, 13 Apr 2013 14:27:35 +0200
To:	James E Keenan via RT <bug-App-ArchiveDevelCover [...] rt.cpan.org>
From:	Thomas Klausner <domm [...] plix.at>

Hi! On Sat, Apr 13, 2013 at 05:47:04AM -0400, James E Keenan via RT wrote: Show quoted text

> Which led me to (finally!) install App-ArchiveDevelCover, about which > I hears you speak at the DC-Baltimore Perl Workshop last April (and > where I'm speaking next weekend).

That was a very nice workshop! And would really like to visit NY again, we had a lovely time there... Show quoted text

> 1. It would be good to include in the documentation a reminder that > you must do 'cover -delete' between runs of Devel::Cover and > archive_devel_cover.pl. Otherwise, the latter program says, "This > coverage report has already been archived."

ok, I wasn't really aware of that. I usually always delete the old coverage report (at least for the project I wrote A-ADC for), because merging coverage takes ages. Show quoted text

> 2. More importantly, archive_devel_cover.pl appears to be hard-wired > to report only statement, sub and total percentages. It omits branch > and condition coverage -- which I rely heavily on -- as well as Pod > and time coverage -- which I personally don't use.

Yes, I decideed to do that this way for a better overview. But I agree on all the reasons to provide why including them might be a good idea. Show quoted text

> So, can you give us an idea as to whether archiving of branch and > condition coverage is feasible and might be forthcoming?

That would be quite easy. I would just need to extract more cols from the coverage table, and add those cols to the report. Of course some kind of user interface/config file would be needed. And it would make a great deal of sense to use the actual database instead of mucking around in the generated HTML... We currently have the first sunny weekend in Vienna (yay!), so I wont spend much time at the laptop. But if you want to submit a patch/pull request, I will most likley merge it. Or I will take a look at refactoring the code next week. If you want to give it a try, please note that the code just grew from a quick hack and thus includes a lot of inlined HTML. And adding a new report is done by splicing a new line into a html table (which makes it rather hard to add a new column) - a better approach would be to rewrite the overview table from scratch each time (which would mean we'd have to parse the old data again..) To sum up: Yes, I'd like to (optionally) include more of the Devel-Cover metrics. Greetings & have a nice hackathon! domm -- #!/usr/bin/perl http://domm.plix.at for(ref bless{},just'another'perl'hacker){s-:+-$"-g&&print$_.$/}

Sat Apr 13 08:27:18 2013 The RT System itself - Status changed from 'new' to 'open'

Sat Apr 13 12:11:03 2013 jkeenan [...] cpan.org - Correspondence added

On Sat Apr 13 08:27:18 2013, domm@plix.at wrote: Show quoted text

> Hi! >

Domm, thanks for your response, which both Paul and I have read. Paul observed that it would probably be easier to pull the coverage results directly from the database underneath cover_db/ than by scraping the HTML. So my new, intermediate goal is to learn how to do that. Let me ask you if the following is a fair characterization of what the program does *from the user's point of view* (i.e., abstracting from implementation details): ########## What does this package actually do? On the first run, it copies the .html files created by a run of Devel::Cover into a subdirectory named 'archive/<DateTime>/' underneath that designated by the value of the 'to' option. Those files include: archive/<DateTime>/coverage.html archive/<DateTime>/blib-lib-Data-Dumper-pm.html archive/<DateTime>/blib-lib-Data-Dumper-pm--branch.html archive/<DateTime>/blib-lib-Data-Dumper-pm--condition.html archive/<DateTime>/blib-lib-Data-Dumper-pm--subroutine.html It also creates the first version of the archive summary .html file: archive/index.html On the second run, it creates another 'archive/<2nd_DateTime>/' subdirectory and copies the 5 .html files into it. It then updates 'archive/index.html' by adding the percentages from the second run to the top of the chart. In addition, it creates the first instance of the diff .html file: archive/<2nd_DateTime>/diff.html On the third and subsequent runs, it presumably (I haven't tried this yet), creates yet another 'archive/<3rd_DateTime>' subdirectory, copies 5 .html files into it, updates 'archive/index.html', and creates the next instance of the diff .html file: archive/<3rd_DateTime>/diff.html # must verify directory So, the program has the potential to store *a lot of* .html files as well as a .diff html file for each run of the program. Only the 'archive/index.html' file appears to be modified in place -- but that is probably the most important output file of all, as it is the one that logs progress over time. ########## Thank you very much. Jim Keenan

Sat Apr 13 13:54:32 2013 domm [...] plix.at - Correspondence added

Subject:	Re: [rt.cpan.org #84605] Request inclusion of branch and condition coverage in archive reports
Date:	Sat, 13 Apr 2013 19:54:58 +0200
To:	James E Keenan via RT <bug-App-ArchiveDevelCover [...] rt.cpan.org>
From:	Thomas Klausner <domm [...] plix.at>

Hi! On Sat, Apr 13, 2013 at 12:11:03PM -0400, James E Keenan via RT wrote: Show quoted text

> Paul observed that it would probably be easier to pull the coverage > results directly from the database underneath cover_db/ than by > scraping the HTML. So my new, intermediate goal is to learn how to do > that.

Yes, Paul already suggested that to me some time ago, but I never got the tuits... Show quoted text

> Let me ask you if the following is a fair characterization of what the > program does *from the user's point of view* (i.e., abstracting from > implementation details):

Yes, this is correct. It also writes the timestamp and the three coverage critieria I care into a file called archive_db. This could be used to regenerate the index html file. If you have any more questions, just ask, though I'm afraid I won't read my email again until tomorrow ~10:00. Greetings, domm -- #!/usr/bin/perl http://domm.plix.at for(ref bless{},just'another'perl'hacker){s-:+-$"-g&&print$_.$/}