Bug #80640 for Pod-Perldoc: perldoc does not display function definitions with the same encoding used in perlfunc.pod

Mon Nov 05 12:31:33 2012 explorer [...] joaquinferrero.com - Ticket created

CC:	explorer [...] joaquinferrero.com
Subject:	perldoc does not display function definitions with the same encoding used in perlfunc.pod

perldoc -L ES perlfunc works because perldoc reads the '=encoding utf8', but perldoc -L ES -f chr won’t render properly characters with diacritic marks: chr NÃ<U+009A>MERO chr Devuelve el carÃ¡cter representado por NÃ<U+009A>MERO en el conjunto de caracteres. Por ejemplo, "chr(65)" es "A" tanto en ASCII como en Unicode, y chr(0x263a) es una cara sonriente en Unicode.

Mon Nov 05 12:39:33 2012 mrallen1 [...] yahoo.com - Correspondence added

On Mon Nov 05 12:31:33 2012, explorer@joaquinferrero.com wrote: Show quoted text

> > perldoc -L ES perlfunc > > works because perldoc reads the '=encoding utf8', but > > perldoc -L ES -f chr > > won’t render properly characters with diacritic marks: > > chr NÃ<U+009A>MERO > chr Devuelve el carÃ¡cter representado por NÃ<U+009A>MERO en > el conjunto > de caracteres. Por ejemplo, "chr(65)" es "A" tanto en > ASCII > como en Unicode, y chr(0x263a) es una cara sonriente > en Unicode. >

Great find. Thanks for the report. We definitely need to fix this. I'm curious if running perldoc with the '-o term' flag makes any difference.

Mon Nov 05 12:39:35 2012 The RT System itself - Status changed from 'new' to 'open'

Mon Jan 28 22:12:10 2013 mrallen1 [...] yahoo.com - Correspondence added

On Mon Nov 05 12:31:33 2012, explorer@joaquinferrero.com wrote: Show quoted text

> > perldoc -L ES perlfunc > > works because perldoc reads the '=encoding utf8', but > > perldoc -L ES -f chr > > won’t render properly characters with diacritic marks: > > chr NÃ<U+009A>MERO > chr Devuelve el carÃ¡cter representado por NÃ<U+009A>MERO en > el conjunto > de caracteres. Por ejemplo, "chr(65)" es "A" tanto en > ASCII > como en Unicode, y chr(0x263a) es una cara sonriente > en Unicode. >

I just uploaded 3.19_01 to CPAN. I would really appreciate it if you'd test that out and let me know if this is still a problem. Thanks.

Tue Feb 05 20:54:56 2013 explorer [...] joaquinferrero.com - Correspondence added

Subject:	Re: [rt.cpan.org #80640] perldoc does not display function definitions with the same encoding used in perlfunc.pod
Date:	Wed, 06 Feb 2013 02:54:38 +0100
To:	bug-Pod-Perldoc [...] rt.cpan.org
From:	Joaquin Ferrero <explorer [...] joaquinferrero.com>

El 29/01/13 04:12, Mark Allen via RT escribió: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=80640 > > > On Mon Nov 05 12:31:33 2012, explorer@joaquinferrero.com wrote:

>> >> perldoc -L ES perlfunc >> >> works because perldoc reads the '=encoding utf8', but >> >> perldoc -L ES -f chr >> >> won’t render properly characters with diacritic marks: >> >> chr NÃ<U+009A>MERO >> chr Devuelve el carÃ¡cter representado por NÃ<U+009A>MERO en >> el conjunto >> de caracteres. Por ejemplo, "chr(65)" es "A" tanto en >> ASCII >> como en Unicode, y chr(0x263a) es una cara sonriente >> en Unicode. >>

> > I just uploaded 3.19_01 to CPAN. I would really appreciate it if you'd > test that out and let me know if this is still a problem. > > Thanks. >

Tested: -----------------------------------------8<----------------------------------------------- $ perldoc -V Perldoc v3.19_01, under perl v5.016000 for linux $ export PERLDOC_POD2=1 $ perldoc -f chmod chmod LISTA Cambia los permisos de una lista de archivos. El primer elemento de la lista debe ser un modo numérico, que normalmente estará en octal, y que, definitivamente, no debe ser una cadena de d\303gitos en octal: 0644 es correcto, "0644" no lo es. Devuelve el número de archivos cambiados con éxito. Vea también "oct" si todo lo que tiene es una cadena. -----------------------------------------8<----------------------------------------------- ¡Oops! The 'í' char (small i latin with acute diacritic mark) is translated to only 0xc3 byte, the preamble of \xc3 \xad actual utf-8 chars, but the second byte is not emitted (at word "dígitos", line three). All rest of special char are correct. I tracked two chars into the perldoc process: 0. perlfunc file, chmod description: é : \xc3 \xa9 í : \xc3 \xad (all pod is utf-8 encoded) 1. perldoc -f chmod. perldoc find and extract the correct segment of chmod function description and write it to /tmp/ file. é : \xc3 \xa9 í : \xc3 \xad (the segment is same content that perlfunc.pod. Only '=over 8' and '=back' are added) 2. perldoc render this file (call to ToTerm.pm) into another file: é : \xc3 \xa9 í : \xc3 (Oops!, the second byte of í are lost!) 3. perldoc execute the pager ... This fail, also: -----------------------------------------8<----------------------------------------------- Show quoted text

> perldoc -o text -d kk2 -f chmod

Perldoc (Pod::Perldoc::ToText) output saved to kk2 -----------------------------------------8<----------------------------------------------- Then, I edited perlfunc.pod and I wrote all special Spanish char: -----------------------------------------8<----------------------------------------------- $ perldoc -f chmod chmod LISTA áÁéÉ303ÍóÓúÚüÜñÑçÇ -----------------------------------------8<----------------------------------------------- Yes! All work, but not the 'í' :) JF

Sat Apr 27 01:36:49 2013 mrallen1 [...] yahoo.com - Correspondence added

On Mon Nov 05 12:31:33 2012, explorer@joaquinferrero.com wrote: Show quoted text

> > perldoc -L ES perlfunc > > works because perldoc reads the '=encoding utf8', but > > perldoc -L ES -f chr > > won’t render properly characters with diacritic marks: > > chr NÃ<U+009A>MERO > chr Devuelve el carÃ¡cter representado por NÃ<U+009A>MERO en > el conjunto > de caracteres. Por ejemplo, "chr(65)" es "A" tanto en > ASCII > como en Unicode, y chr(0x263a) es una cara sonriente > en Unicode. >

Believe this is fixed in Pod::Perldoc 3.20.

Wed May 15 18:50:48 2013 explorer [...] joaquinferrero.com - Correspondence added

El Sáb Abr 27 01:36:49 2013, mallen escribió: Show quoted text

> Believe this is fixed in Pod::Perldoc 3.20.

Perl v5.18 only have v3.19 :(

Wed May 15 23:18:54 2013 mrallen1 [...] yahoo.com - Correspondence added

On Wed May 15 18:50:48 2013, explorer@joaquinferrero.com wrote: Show quoted text

> El Sáb Abr 27 01:36:49 2013, mallen escribió:

> > Believe this is fixed in Pod::Perldoc 3.20.

> > Perl v5.18 only have v3.19 > > :( >

True, sorry about that - I asked RJBS about getting it in before 5.18 ships, but it was too close to the release date. Fortunately, the module can be installed right off CPAN and it will *definitely* be in 5.20.

Sun May 19 21:44:19 2013 http://joaquinferrero.com/ - Correspondence added

From:

Joaquin Ferrero <explorer [...] joaquinferrero.com>

I installed Perl v5.18 and Pod::Perldoc v3.20 from CPAN. Problems: 1.- Now, Pod::Perldoc use Pod::Perldoc::ToTerm formatter: Perldoc.pm: 473 $self->opt_o_with('term') unless $self->is_mswin32 || $self->is_dos and the width of screen is taken from Text.pm: 121 $$self{opt_width} = 76 unless defined $$self{opt_width}; and more later: Text/Termcap.pm: 69 unless (defined $$self{width}) { 70 $$self{opt_width} = $ENV{COLUMNS} || $$term{_co} || 80; 71 $$self{opt_width} -= 2; 72 } a.- What is $$self{width}? It is not used anymore... Maybe 'opt_width'? but... Text.pm initialized $$self{opt_width}, ever b.- $$self{opt_width} is initialized with $ENV{COLUMNS} or $$term{_co}. My screen (tmux, TERM=screen) have $COLUMNS environment variable but is not exported (I will need to write "export COLUMNS" into .bashrc file), so $ENV{COLUMNS} is void. Then, $$self{opt_width} is set with the $$term{_co} value (80 columns, ever, because is a capability (Term::Cap)). Result: the pod is formatted to 80-2 columns, ever. At Perl v5.14, Pod::Perldoc::ToMan used `stty -a` to get width screen. If I change $$self{width} with $$self{opt_width}, this line work: $ perldoc -w width:$[COLUMNS-2] perldata If I leave $$self{width}, this work: $ export COLUMNS $ perldoc perldata Best solution I found: A.- Comment or remove line 121 of Text.pm (don't initialize $$self{opt_width}) B.- Change $$self{width} with $$self{opt_width} at Text/Termcap.pm C.- $ export COLUMNS (or in .bashrc) D.- $ perldoc perldata 2.- I suspect that is a error: Perldoc/ToTerm.pm: 36 and print "About to call new Pod::Text::Termcap ", 37 $Pod::Text::VERSION ? "(v$Pod::Text::VERSION) " : '', 38 "with options: ", 39 @options ? "[@options]" : "(nil)", "\n"; 40 ; 41 42 Pod::Text::Termcap->new(@options)->parse_from_file(@_); The message show the value of $Pod::Text::VERSION, not $Pod::Text::Termcap::VERSION Problems solved: 1.- Rendering utf8 pod are Ok below Pod::Perldoc::ToTerm formatter: $ perldoc -L ES perlfunc NOMBRE perlfunc - Funciones predefinidas en Perl DESCRIPCIÓN Las funciones de esta sección pueden servir como términos en una expresión. Se agrupan en dos categorías principales: operadores de listas 2.- perldoc -f <function> work now Ok with utf8 pods: $ perldoc -L ES -f chr chr NÚMERO chr Devuelve el carácter representado por NÚMERO en el conjunto de caracteres. Por ejemplo, "chr(65)" es "A" tanto en ASCII como en Unicode, y chr(0x263a) es una cara sonriente en Unicode.

Mon May 20 10:20:26 2013 mrallen1 [...] yahoo.com - Correspondence added

Subject:	Re: [rt.cpan.org #80640] perldoc does not display function definitions with the same encoding used in perlfunc.pod
Date:	Mon, 20 May 2013 07:20:09 -0700 (PDT)
To:	"bug-Pod-Perldoc [...] rt.cpan.org" <bug-Pod-Perldoc [...] rt.cpan.org>
From:	Mark Allen <mrallen1 [...] yahoo.com>

I'd like to open a new ticket for this particular issue. Thanks for reporting it. Show quoted text

________________________________ From: http://joaquinferrero.com/ via RT <bug-Pod-Perldoc@rt.cpan.org> To: Sent: Sunday, May 19, 2013 8:44 PM Subject: [rt.cpan.org #80640] perldoc does not display function definitions with the same encoding used in perlfunc.pod Queue: Pod-Perldoc Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=80640 > I installed Perl v5.18 and Pod::Perldoc v3.20 from CPAN. Problems: 1.- Now, Pod::Perldoc use Pod::Perldoc::ToTerm formatter: Perldoc.pm: 473 $self->opt_o_with('term') unless $self->is_mswin32 || $self->is_dos and the width of screen is taken from Text.pm: 121 $$self{opt_width} = 76 unless defined $$self{opt_width}; and more later: Text/Termcap.pm: 69 unless (defined $$self{width}) { 70 $$self{opt_width} = $ENV{COLUMNS} || $$term{_co} || 80; 71 $$self{opt_width} -= 2; 72 } a.- What is $$self{width}? It is not used anymore... Maybe 'opt_width'? but... Text.pm initialized $$self{opt_width}, ever b.- $$self{opt_width} is initialized with $ENV{COLUMNS} or $$term{_co}. My screen (tmux, TERM=screen) have $COLUMNS environment variable but is not exported (I will need to write "export COLUMNS" into .bashrc file), so $ENV{COLUMNS} is void. Then, $$self{opt_width} is set with the $$term{_co} value (80 columns, ever, because is a capability (Term::Cap)). Result: the pod is formatted to 80-2 columns, ever. At Perl v5.14, Pod::Perldoc::ToMan used `stty -a` to get width screen. If I change $$self{width} with $$self{opt_width}, this line work: $ perldoc -w width:$[COLUMNS-2] perldata If I leave $$self{width}, this work: $ export COLUMNS $ perldoc perldata Best solution I found: A.- Comment or remove line 121 of Text.pm (don't initialize $$self{opt_width}) B.- Change $$self{width} with $$self{opt_width} at Text/Termcap.pm C.- $ export COLUMNS (or in .bashrc) D.- $ perldoc perldata 2.- I suspect that is a error: Perldoc/ToTerm.pm: 36 and print "About to call new Pod::Text::Termcap ", 37 $Pod::Text::VERSION ? "(v$Pod::Text::VERSION) " : '', 38 "with options: ", 39 @options ? "[@options]" : "(nil)", "\n"; 40 ; 41 42 Pod::Text::Termcap->new(@options)->parse_from_file(@_); The message show the value of $Pod::Text::VERSION, not $Pod::Text::Termcap::VERSION Problems solved: 1.- Rendering utf8 pod are Ok below Pod::Perldoc::ToTerm formatter: $ perldoc -L ES perlfunc NOMBRE perlfunc - Funciones predefinidas en Perl DESCRIPCIÓN Las funciones de esta sección pueden servir como términos en una expresión. Se agrupan en dos categorías principales: operadores de listas 2.- perldoc -f <function> work now Ok with utf8 pods: $ perldoc -L ES -f chr chr NÚMERO chr Devuelve el carácter representado por NÚMERO en el conjunto de caracteres. Por ejemplo, "chr(65)" es "A" tanto en ASCII como en Unicode, y chr(0x263a) es una cara sonriente en Unicode.

Mon May 20 11:33:45 2013 mrallen1 [...] yahoo.com - Status changed from 'open' to 'resolved'

Mon May 20 11:33:45 2013 mrallen1 [...] yahoo.com - Fixed in 3.20 added