Bug #78043 for Pod-Tidy: data paragraphs in PODs are mangled on tidy

Mon Jun 25 16:48:16 2012 finn [...] timeghost.net - Ticket created

Subject:

data paragraphs in PODs are mangled on tidy

Occasionally data paragraphs in PODs are mangled during a tidying run. This is due to how the POD::Tidy module parses and cleans PODs in the codebase. Long story short: Pod::Tidy should be ignoring everything in between =begin and =end, (data paragraphs) and leaving them as-is instead of wrapping them, but it isn't. AFAICT it is not looking at or detecting the existence of of those format blocks. I went through the code of Pod::Tidy (which subclasses Pod::Wrap which subclasses Pod::Parser) and in all three of those modules I couldn't find any special cases for data paragraphs. It seems like it should be the responsibility of Pod::Wrap and Pod::Tidy to detect that a paragraph of text is a data paragraph and not perform any tidying on it. But these modules are not examining the parse tree or anything else beyond whether a paragraph is a verbatim or a textblock. So what is happening is it is seeing chunks of text as textblocks (not double- checking to see if they are data paragraphs) and wraps them even when it shouldn't. We discovered this because we had sample pieces of JSON in our own =begin and =end data blocks which were then reformatted during a tidy run. http://perldoc.perl.org/perlpodspec.html#About-Data-Paragraphs-and- %22%3dbegin%2f%3dend%22-Regions "Data paragraphs are typically used for inlining non-Pod data that is to be used (typically passed through) when rendering the document to a specific format"

Mon Jun 25 21:10:30 2012 MLAWREN [...] cpan.org - Correspondence added

Finn, do you have a test case? I can't reproduce this trivially here. (btw I am not the maintainer of Pod::Tidy, I'm just asking out of interest so ignore that question unless you think it will help this ticket) A possible work around may be to indent the text between the =begin and =end blocks, to make the parser treat it as code. Mark.

Mon Jun 25 21:10:31 2012 The RT System itself - Status changed from 'new' to 'open'

Mon Jun 25 21:24:26 2012 jhoblitt [...] cpan.org - Correspondence added

Hello, On Mon Jun 25 16:48:16 2012, finn wrote: Show quoted text

> Occasionally data paragraphs in PODs are mangled during a tidying run.

Is the same paragraph sometimes treated differently between runs? Wow. That's scary. Show quoted text

> Long story short: Pod::Tidy should be ignoring everything in between > =begin and =end, (data > paragraphs) and leaving them as-is instead of wrapping them, but it > isn't.

Well, between =begin <identifier> / =end <identifier> blocks. I agree the behavior your reporting is a bug. Show quoted text

> We discovered this because we had sample pieces of JSON in our own > =begin and =end data > blocks which were then reformatted during a tidy run.

Would you be willing to write a test case or at least submit a section of test pod? What platform are you on? Do the tests all pass? Cheers, -Josh

Tue Jun 26 15:30:52 2012 finn [...] timeghost.net - Correspondence added

On Mon Jun 25 21:10:30 2012, MLAWREN wrote: Show quoted text

> Finn, do you have a test case? I can't reproduce this trivially here. > (btw I am not the maintainer of Pod::Tidy, I'm just asking out of > interest so ignore that question unless you think it will help this ticket) > > A possible work around may be to indent the text between the =begin and > =end blocks, to make the parser treat it as code.

Attached is a before (list.pl) and after (list-pod-tidied.pl) file which shows some lines getting reformatted via rewrapping when they shouldn't be. The expectation is that the output of running podtidy on list.pl would be identical to the original contents of list.pl. Instead, the output is what is seen in list-pod- tidied.pl. Indenting the appropriate sections is actually the workaround we came up with as well. -F

Subject:

list-pod-tidied.pl

=head3 B<list> Return a list of data. =begin return_value [{ "id": 3, "start_date": "2011-10-01 00:00:00", "end_date": "2011-10-03 00:00:00", "name": "A Name", }, { "id": 5, "start_date": "2011-09-01 00:00:00", "end_date": "2011-09-03 00:00:00", "name": "Another Name", }] =end return_value =cut sub list { my $data = <<'JSON'; [{ "id": 3, "start_date": "2011-10-01 00:00:00", "end_date": "2011-10-03 00:00:00", "name": "A Name", }, { "id": 5, "start_date": "2011-09-01 00:00:00", "end_date": "2011-09-03 00:00:00", "name": "Another Name", }] JSON return $data; }

Subject:

list.pl

=head3 B<list> Return a list of data. =begin return_value [{ "id": 3, "start_date": "2011-10-01 00:00:00", "end_date": "2011-10-03 00:00:00", "name": "A Name", }, { "id": 5, "start_date": "2011-09-01 00:00:00", "end_date": "2011-09-03 00:00:00", "name": "Another Name", }] =end return_value =cut sub list { my $data = <<'JSON'; [{ "id": 3, "start_date": "2011-10-01 00:00:00", "end_date": "2011-10-03 00:00:00", "name": "A Name", }, { "id": 5, "start_date": "2011-09-01 00:00:00", "end_date": "2011-09-03 00:00:00", "name": "Another Name", }] JSON return $data; }

Tue Jun 26 16:20:18 2012 finn [...] timeghost.net - Correspondence added

Show quoted text

> > Occasionally data paragraphs in PODs are mangled during a tidying run.

> > Is the same paragraph sometimes treated differently between runs? Wow. > That's scary.

Sorry, I should have been more the specific. The same paragraphs are treated the same in each run. The problem is that the data paragraphs should be left alone, and they aren't. Show quoted text

> Would you be willing to write a test case or at least submit a section > of test pod? > > What platform are you on? Do the tests all pass?

I submitted a couple of files in my previous message which have some sample POD in them. Let me know if that looks ok. Currently running on recent versions of Mac OS X (Lion or Snow Leopard). perl version: v5.10.1 (*) built for darwin-2level In this environment using Pod::Tidy 0.10 I had thought all the tests were passing, but on examination, t/06_pod_tidy_process_pod_queue.t and t/08_pod_tidy_tidy_files.t both hang. t/06_pod_tidy_process_pod_queue.t hangs at the line: my $processed = Pod::Tidy::process_pod_queue( queue => [$tmp_valid->filename], ); and t/08_pod_tidy_tidy_files.t hangs at the line: my $status = Pod::Tidy::tidy_files( files => [$tmp_valid->filename, $tmp_invalid->filename], ); I haven't dug in further into why they are doing so.

Fri Aug 03 00:43:17 2012 jhoblitt [...] cpan.org - Correspondence added

I don't presently have access to a lion test system I can't help redo the issue your issue but I suspect the issue must be fairly fundimental. Has that been resolved yet? I've checked your test case into the github repo. https://github.com/jhoblitt/Pod-Tidy/commit/8d917788d081d0cf84130ffb0a4996923814dce6 The bug is in Pod::Wrap::Pretty, which is a subclass of Pod::Wrap, which is itself apparently a subclass of Pod::Parser. It's been long enough that I had forgotten the details of the code. I don't think the fix will be that difficult but I won't have time to try to sort before this weekend at the earlist. -Josh