Bug #77465 for Pod-Perldoc: Forcing UTF8 causes formatting issues on slightly older groffs

Sun May 27 03:30:30 2012 c.nehren [...] shadowcat.co.uk - Ticket created

Subject:	Forcing UTF8 causes formatting issues on slightly older groffs
Date:	Sun, 27 May 2012 03:30:15 -0400
To:	bug-pod-perldoc [...] rt.cpan.org
From:	Chris Nehren <c.nehren [...] shadowcat.co.uk>

Not all systems have a bleeding edge groff available, and not all systems are capable of handling UTF8 yet. This means the unconditional forcing of the utf8 flag in lib/Pod/Perldoc/ToMan.pm line 141 causes formatting issues. This looks like a remnant of a commit bdfoy made some years ago that also unconditionally set the UTF8 flag (without checking *at all* that the system was actually capable of dealing with the results). This either needs to be wrapped in some conditionals or removed. It looks like the code tries to do this but doesn't actually do a very good job of it. With that line present, the current Pod-Perldoc git HEAD renders this: PERL(1) User Contributed Perl Documentation PERL(1) NAME perl - The Perl 5 language interpreter SYNOPSIS perl [<C2> -sTtuUWX<C2> ] [<C2> -hv<C2> ]<C2> [<C2> -V[:configvar]<C2> ] [<C2> -cw<C2> ]<C2> [<C2> -d[t][:debugger]<C2> ]<C2> [<C2> -D[number/list]<C2> ] [<C2> -pna<C2> ]<C2> [<C2> -Fpattern<C2> ]<C2> [<C2> -l[octal]<C2> ]<C2> [<C2> -0[octal/hexadecimal]<C2> ] [<C2> -Idir<C2> ]<C2> [<C2> -m[-]module<C2> ]<C2> [<C2> -M[-]'module...'<C2> ]<C2> [<C2> -f<C2> ] [<C2> -C<C2> [number/list]<C2> ] [<C2> -S<C2> ] [<C2> -x[dir]<C2> ] [<C2> -i[extension]<C2> ] [<C2> [-e|-E]<C2> 'command'<C2> ]<C2> [<C2> --<C2> ]<C2> [<C2> programfile<C2> ]<C2> [<C2> argument<C2> ]... If I comment out that line, I get the much nicer: PERL(1) User Contributed Perl Documentation PERL(1) NAME perl - The Perl 5 language interpreter SYNOPSIS perl [ -sTtuUWX ] [ -hv ] [ -V[:configvar] ] [ -cw ] [ -d[t][:debugger] ] [ -D[number/list] ] [ -pna ] [ -Fpattern ] [ -l[octal] ] [ -0[octal/hexadecimal] ] [ -Idir ] [ -m[-]module ] [ -M[-]'module...' ] [ -f ] [ -C [number/list] ] [ -S ] [ -x[dir] ] [ -i[extension] ] [ [-e|-E] 'command' ] [ -- ] [ programfile ] [ argument ]... I will note that my system is quite capable of handling UTF-8 (have a snowman: ☃), and therefore it's my (slightly) out of date groff that is the issue. I have confirmed this by installing a new groff, uncommenting the problematic line, and running perldoc with -n $PATH_TO_NEW_GROFF. I would rather not install a separate groff (and its dependencies) to cope with a buggy Perl library that can be fixed by removing one erroneous line of code. -- Chris Nehren | Coder, Sysadmin, Masochist Shadowcat Systems Ltd. | http://shadowcat.co.uk/

Sat Oct 20 17:46:04 2012 jkeenan [...] cpan.org - Reference to https://rt.perl.org/rt3/Ticket/Display.html?id=113406 added

Sat Oct 20 17:53:42 2012 jkeenan [...] cpan.org - Correspondence added

On Sun May 27 03:30:30 2012, c.nehren@shadowcat.co.uk wrote: Show quoted text

> Not all systems have a bleeding edge groff available, and not all > systems are capable of handling UTF8 yet. This means the unconditional > forcing of the utf8 flag in lib/Pod/Perldoc/ToMan.pm line 141 causes > formatting issues. This looks like a remnant of a commit bdfoy made > some > years ago that also unconditionally set the UTF8 flag (without > checking > *at all* that the system was actually capable of dealing with the > results). This either needs to be wrapped in some conditionals or > removed. It looks like the code tries to do this but doesn't actually > do > a very good job of it. > > With that line present, the current Pod-Perldoc git HEAD renders this: > > PERL(1) User Contributed Perl Documentation > PERL(1) > > > > NAME > perl - The Perl 5 language interpreter > > SYNOPSIS > perl [<C2> -sTtuUWX<C2> ] [<C2> -hv<C2> ]<C2> [<C2> > -V[:configvar]<C2> ] > [<C2> -cw<C2> ]<C2> [<C2> -d[t][:debugger]<C2> ]<C2> [<C2> > -D[number/list]<C2> ] > [<C2> -pna<C2> ]<C2> [<C2> -Fpattern<C2> ]<C2> [<C2> > -l[octal]<C2> ]<C2> [<C2> -0[octal/hexadecimal]<C2> ] > [<C2> -Idir<C2> ]<C2> [<C2> -m[-]module<C2> ]<C2> [<C2> > -M[-]'module...'<C2> ]<C2> [<C2> -f<C2> ] > [<C2> -C<C2> [number/list]<C2> ] [<C2> -S<C2> ] > [<C2> -x[dir]<C2> ] > [<C2> -i[extension]<C2> ] > [<C2> [-e|-E]<C2> 'command'<C2> ]<C2> [<C2> --<C2> ]<C2> > [<C2> programfile<C2> ]<C2> [<C2> argument<C2> ]... > > > If I comment out that line, I get the much nicer: > > PERL(1) User Contributed Perl Documentation > PERL(1) > > > > NAME > perl - The Perl 5 language interpreter > > SYNOPSIS > perl [ -sTtuUWX ] [ -hv ] [ -V[:configvar] ] > [ -cw ] [ -d[t][:debugger] ] [ -D[number/list] ] > [ -pna ] [ -Fpattern ] [ -l[octal] ] [ > -0[octal/hexadecimal] ] > [ -Idir ] [ -m[-]module ] [ -M[-]'module...' ] [ -f ] > [ -C [number/list] ] [ -S ] [ -x[dir] ] > [ -i[extension] ] > [ [-e|-E] 'command' ] [ -- ] [ programfile ] [ argument > ]... > > I will note that my system is quite capable of handling UTF-8 (have a > snowman: ☃), and therefore it's my (slightly) out of date groff that > is > the issue. I have confirmed this by installing a new groff, > uncommenting > the problematic line, and running perldoc with -n $PATH_TO_NEW_GROFF. > I > would rather not install a separate groff (and its dependencies) to > cope > with a buggy Perl library that can be fixed by removing one erroneous > line of code. >

I confirm Chris's findings. I filed RT #113406 in the Perl 5 bug queue back in May; it's still open. When I first noted the problem, I configured, built and installed the latest groff (1.21) from source. The problem didn't go away because I didn't know about the 'perldoc -n </path/to/better/groff> setting Chris presented above. But I don't want to have to do that every time I use 'perldoc'. Let's get this bug fixed! Thank you very much. Jim Keenan

Sat Oct 20 17:53:43 2012 The RT System itself - Status changed from 'new' to 'open'

Sat Oct 20 18:12:39 2012 jkeenan [...] cpan.org - Correspondence added

On Sat Oct 20 17:53:42 2012, JKEENAN wrote: Show quoted text

> > I confirm Chris's findings. I filed RT #113406 in the Perl 5 bug queue > back in May; it's still open. When I first noted the problem, I > configured, built and installed the latest groff (1.21) from source. > The problem didn't go away because I didn't know about the 'perldoc -n > </path/to/better/groff> setting Chris presented above. But I don't want > to have to do that every time I use 'perldoc'. >

I was too optimistic. [perl] 519 $ which groff /usr/local/bin/groff [perl] 520 $ groff --version GNU groff version 1.21 [snip] called subprograms: GNU grops (groff) version 1.21 GNU troff (groff) version 1.21 [perl] 521 $ perldoc -n /usr/local/bin/groff perl PERL(1) User Contributed Perl Documentation PERL(1) NAME perl - The Perl 5 language interpreter SYNOPSIS perl [<C2><A0>-sTtuUWX<C2><A0>] [<C2><A0>-hv<C2><A0>]<C2><A0>[<C2> <A0>-V[:configvar]<C2><A0>] [<C2><A0>-cw<C2><A0>]<C2><A0>[<C2><A0>-d[t][:debugger]<C2><A0>]<C2> <A0>[<C2><A0>-D[number/list]<C2><A0>] [<C2><A0>-pna<C2><A0>]<C2><A0>[<C2><A0>-Fpattern<C2><A0>]<C2><A0>[ <C2><A0>-l[octal]<C2><A0>]<C2><A0>[<C2><A0>-0[octal/hexadecimal]<C2><A0>] [<C2><A0>-Idir<C2><A0>]<C2><A0>[<C2><A0>-m[-]module<C2><A0>]<C2><A0> [<C2><A0>-M[-]'module...'<C2><A0>]<C2><A0>[<C2><A0>-f<C2><A0>] [<C2><A0>-C<C2><A0>[number/list]<C2><A0>] [<C2><A0>-S<C2><A0>] [<C2><A0>-x[dir]<C2><A0>] [<C2><A0>-i[extension]<C2><A0>] [<C2><A0>[-e|-E]<C2><A0>'command'<C2><A0>]<C2><A0>[<C2><A0>--<C2> <A0>]<C2><A0>[<C2><A0>programfile<C2><A0>]<C2><A0>[<C2><A0>argument<C2><A0>]... So the problem is worse than I thought.

Mon Jan 14 19:44:36 2013 jkeenan [...] cpan.org - Correspondence added

On Sat Oct 20 18:12:39 2012, JKEENAN wrote: Show quoted text

> On Sat Oct 20 17:53:42 2012, JKEENAN wrote:

> >

Show quoted text

> > [perl] 519 $ which groff > /usr/local/bin/groff > [perl] 520 $ groff --version > GNU groff version 1.21 > [snip] > called subprograms: > > GNU grops (groff) version 1.21 > GNU troff (groff) version 1.21 > [perl] 521 $ perldoc -n /usr/local/bin/groff perl > > PERL(1) User Contributed Perl Documentation > PERL(1) > > > > NAME > perl - The Perl 5 language interpreter > > SYNOPSIS > perl [<C2><A0>-sTtuUWX<C2><A0>] > [<C2><A0>-hv<C2><A0>]<C2><A0>[<C2> > <A0>-V[:configvar]<C2><A0>] > > [<C2><A0>-cw<C2><A0>]<C2><A0>[<C2><A0>-d[t][:debugger]<C2><A0>]<C2> > <A0>[<C2><A0>-D[number/list]<C2><A0>] > > [<C2><A0>-pna<C2><A0>]<C2><A0>[<C2><A0>-Fpattern<C2><A0>]<C2><A0>[ > <C2><A0>-l[octal]<C2><A0>]<C2><A0>[<C2><A0>- > 0[octal/hexadecimal]<C2><A0>] > > [<C2><A0>-Idir<C2><A0>]<C2><A0>[<C2><A0>-m[-]module<C2><A0>]<C2><A0> > [<C2><A0>-M[-]'module...'<C2><A0>]<C2><A0>[<C2><A0>-f<C2><A0>] > [<C2><A0>-C<C2><A0>[number/list]<C2><A0>] > [<C2><A0>-S<C2><A0>] > [<C2><A0>-x[dir]<C2><A0>] > [<C2><A0>-i[extension]<C2><A0>] > > [<C2><A0>[-e|-E]<C2><A0>'command'<C2><A0>]<C2><A0>[<C2><A0>--<C2> >

<A0>]<C2><A0>[<C2><A0>programfile<C2><A0>]<C2><A0>[<C2><A0>argument<C2><A0>]... Show quoted text

> > So the problem is worse than I thought.

Pod-Perldoc maintainers: Is there any hope that this problem will be fixed in time for Perl 5.18? Thank you very much. Jim Keenan

Wed Jan 16 21:48:20 2013 mrallen1 [...] yahoo.com - Correspondence added

Show quoted text

> Pod-Perldoc maintainers: Is there any hope that this problem will be > fixed in time for Perl 5.18?

Yes, that's the plan. I hope to have a new version which closes all (or many) tickets ready to put in blead around February 1st-ish. Thanks.

Sun Jan 27 19:58:00 2013 jkeenan [...] cpan.org - Cc JKEENAN added

Sun Jan 27 23:00:28 2013 mrallen1 [...] yahoo.com - Correspondence added

On Mon Jan 14 19:44:36 2013, JKEENAN wrote: Show quoted text

> Pod-Perldoc maintainers: Is there any hope that this problem will be > fixed in time for Perl 5.18?

Committed in 6e1541d. I will be releasing 3.18 to CPAN shortly. Thanks.

Sun Jan 27 23:00:47 2013 mrallen1 [...] yahoo.com - Status changed from 'open' to 'patched'

Sun Jan 27 23:17:17 2013 c.nehren [...] shadowcat.co.uk - Correspondence added

Subject:	Re: [rt.cpan.org #77465] Forcing UTF8 causes formatting issues on slightly older groffs
Date:	Sun, 27 Jan 2013 23:18:37 -0500
To:	Mark Allen via RT <bug-Pod-Perldoc [...] rt.cpan.org>
From:	Chris Nehren <c.nehren [...] shadowcat.co.uk>

On Sun, Jan 27, 2013 at 23:00:29 -0500 , Mark Allen via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=77465 > > > On Mon Jan 14 19:44:36 2013, JKEENAN wrote:

> > Pod-Perldoc maintainers: Is there any hope that this problem will be > > fixed in time for Perl 5.18?

> > Committed in 6e1541d. I will be releasing 3.18 to CPAN shortly.

\o/ Thank you! I've tested this and it does indeed render things as I'd expect. For reference, in case you are curious: GNU groff version 1.19.2 Copyright (C) 2004 Free Software Foundation, Inc. GNU groff comes with ABSOLUTELY NO WARRANTY. You may redistribute copies of groff and its subprograms under the terms of the GNU General Public License. For more information about these matters, see the file named COPYING. called subprograms: GNU grops (groff) version 1.19.2 GNU troff (groff) version 1.19.2 This is the version where I see problms. This is as-shipped for both FreeBSD 9.1 (9.0 when I first filed the bug, I think) and OS X 10.8.2. -- Chris Nehren Shadowcat Alumnus

Sun Jan 27 23:17:18 2013 The RT System itself - Status changed from 'patched' to 'open'

Mon Jan 28 11:32:18 2013 mrallen1 [...] yahoo.com - Correspondence added

Great. Thanks for the followup.

Mon Jan 28 11:32:19 2013 mrallen1 [...] yahoo.com - Status changed from 'open' to 'resolved'

Mon Jan 28 20:23:25 2013 jkeenan [...] cpan.org - Correspondence added

Unfortunately, this doesn't solve *all* the problems I've seen with respect to perldoc. However, in the interest of giving the maintainers a respite and of moving the discussion forward, I'll open a separate ticket once I've had time to assess the remaining problems. Thanks for all your work to date. Jim Keenan

Mon Jan 28 20:23:26 2013 The RT System itself - Status changed from 'resolved' to 'open'

Mon Jan 28 22:03:32 2013 mrallen1 [...] yahoo.com - Correspondence added

On Mon Jan 28 20:23:25 2013, JKEENAN wrote: Show quoted text

> Unfortunately, this doesn't solve *all* the problems I've seen with > respect to perldoc. However, in the interest of giving the maintainers > a respite and of moving the discussion forward, I'll open a separate > ticket once I've had time to assess the remaining problems. Thanks for > all your work to date.

OK. I'm already aware of some of the UTF8 problems reported by the Spanish translation team. Are there other problems? By the way, I just uploaded a developer release of Perldoc which replaces the default formatter (Pod::Man on Unix/Linux) with Pod::Test::Termcap. The default formatter for Win32 and VMS remains Pod::Text. My experience trying to finagle Pod::Man to reliably output UTF8 is vastly complicated by the interaction between perl itself, any given terminal program, Pod::Man, and the nroff implementation on a given Unix flavor.

Mon May 20 11:28:03 2013 mrallen1 [...] yahoo.com - Status changed from 'open' to 'resolved'

Mon May 20 11:28:03 2013 mrallen1 [...] yahoo.com - Fixed in 3.19 added