Bug #59832 for Term-ReadLine-Gnu: clobbers binmode layers on filehandles

Thu Jul 29 14:28:48 2010 CARNIL [...] cpan.org - Ticket created

Subject:	clobbers binmode layers on filehandles
Date:	Thu, 29 Jul 2010 20:28:35 +0200
To:	bug-Term-ReadLine-Gnu [...] rt.cpan.org
From:	Salvatore Bonaccorso <carnil [...] cpan.org>

Hi In Debian we recieved the following bugreport. Could you have a look at it? Many thanks in advance Bests Salvatore ------------------------------------------------------------------------ From: Andrew Pimlott <andrew@pimlott.net> To: Debian Bug Tracking System <submit@bugs.debian.org> Subject: libterm-readline-gnu-perl: clobbers binmode layers on filehandles Date: Thu, 17 Sep 2009 13:31:24 -0700 Package: libterm-readline-gnu-perl Version: 1.19-1 Severity: normal If I call Term::ReadLine->new('test', \*IN, \*OUT); any encoding layers I have set on OUT seem to be removed. I tracked it down to the line in the perl source that sets $Attribs{outstream}. I looked briefly at the .xs source for _rl_store_iostream, but I can't see what's causing this. I hope someone who knows perl internals can figure it out. Here's a complete test program. I run it on a utf-8 terminal. The first print works correctly (the correct utf-8 byte sequence is output), and the second shows a unicode unknown character (U+fffd), because perl outputs the byte \xf3. use Term::ReadLine; binmode(STDOUT, ':encoding(utf-8)'); print STDOUT ">", chr(0xf3), "<\n"; $Term::ReadLine::Gnu::Attribs{outstream} = \*STDOUT; print STDOUT ">", chr(0xf3), "<\n"; Andrew -- System Information: Debian Release: squeeze/sid APT prefers unstable APT policy: (500, 'unstable') Architecture: i386 (i686) Kernel: Linux 2.6.30-1-686 (SMP w/1 CPU core) Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/bash Versions of packages libterm-readline-gnu-perl depends on: ii libc6 2.9-26 GNU C Library: Shared libraries ii libncurses5 5.7+20090803-2 shared libraries for terminal hand ii libreadline5 5.2-6 GNU readline and history libraries ii perl 5.10.0-25 Larry Wall's Practical Extraction ii perl-base [perlapi-5.10.0 5.10.0-25 minimal Perl system libterm-readline-gnu-perl recommends no packages. libterm-readline-gnu-perl suggests no packages. -- no debconf information ------------------------------------------------------------------------

Download signature.asc
application/pgp-signature 835b

Message body not shown because it is not plain text.

Sun Aug 15 10:09:23 2010 HAYASHI [...] cpan.org - Correspondence added

RT-Send-CC:

carnil [...] cpan.org

Hi, Show quoted text

> If I call > > Term::ReadLine->new('test', \*IN, \*OUT); > > any encoding layers I have set on OUT seem to be removed. I tracked

Term::Gnu::ReadLine "is an implementation of Term::ReadLine using the GNU Readline/History Library." as described in the document. The GNU Readline Library may call low level APIs directly. So this is not a bug. BTW if the reporter wants to handle multibyte (ex. UTF-8) characters, the GNU Readline Library support them and, as the result, Term::Gnu::ReadLine also support them. Refer the documents for the GNU Readline Library for details. Show quoted text

> it > down to the line in the perl source that sets $Attribs{outstream}. I > looked briefly at the .xs source for _rl_store_iostream, but I can't > see > what's causing this. I hope someone who knows perl internals can > figure > it out.

Here is a comment in Gnu.pm; --------------------------------------- # Each variable in the GNU Readline Library is tied to an entry of # this hash (%Attribs). By accessing the hash entry, you can read # and/or write the variable in the GNU Readline Library. See the # package definition of Term::ReadLine::Gnu::Var and following code # for more details. --------------------------------------- Refer the chapter 'Tied Variables' in Perl manual, or the following URL; http://perldoc.perl.org/perltie.html I hope this helps. Thanks.

Sun Aug 15 10:09:27 2010 The RT System itself - Status changed from 'new' to 'open'

Sun Aug 15 10:09:28 2010 HAYASHI [...] cpan.org - Status changed from 'open' to 'rejected'

Wed Aug 18 07:23:38 2010 CARNIL [...] cpan.org - Correspondence added

Thanks! Bests Salvatore

Wed Aug 18 07:23:42 2010 The RT System itself - Status changed from 'rejected' to 'open'

Wed Aug 18 07:24:05 2010 CARNIL [...] cpan.org - Status changed from 'open' to 'rejected'

Wed Aug 18 16:36:11 2010 andrew [...] pimlott.net - Correspondence added

CC:	Hiroo_HAYASHI via RT <bug-Term-ReadLine-Gnu [...] rt.cpan.org>, CARNIL [...] cpan.org
Subject:	Re: Bug#547231 closed by Salvatore Bonaccorso <salvatore.bonaccorso@gmail.com> ([rt.cpan.org #59832] clobbers binmode layers on filehandles)
Date:	Wed, 18 Aug 2010 13:35:56 -0700
To:	547231 <547231 [...] bugs.debian.org>
From:	Andrew Pimlott <andrew [...] pimlott.net>

Salvatore, thanks for following up on this report. I have some questions below. Excerpts from owner's message of Wed Aug 18 04:30:08 -0700 2010: Show quoted text

> This is an automatic notification regarding your Bug report

... Show quoted text

> Hi Andrew > > I'm forwarding you the reply from upstream according to your report. > > From: Hiroo_HAYASHI via RT <bug-Term-ReadLine-Gnu@rt.cpan.org> >

> > If I call > > > > Term::ReadLine->new('test', \*IN, \*OUT); > > > > any encoding layers I have set on OUT seem to be removed. I tracked

> > Term::Gnu::ReadLine "is an implementation of Term::ReadLine using the > GNU Readline/History Library." as described in the document. > The GNU Readline Library may call low level APIs directly. So this is > not a bug.

I realize that Readline has to do some low-level things. But I don't understand why it has to clobber the encoding layers on the filehandle. Is it possible to preserve them? Do you not agree that the test program I sent is terribly confusing? If Term::ReadLine::Gnu changes the behavior of the filehandles like this, it should at least be documented. Show quoted text

> BTW if the reporter wants to handle multibyte (ex. UTF-8) characters, > the GNU Readline Library support them and, as the result, > Term::Gnu::ReadLine also support them. Refer the documents for the GNU > Readline Library for details.

I couldn't find the documentation you are refering to, or how I'm supposed to handle multibyte characters with Term::ReadLine::GNU in Perl. I notice that if I set the encoding layer back to UTF-8, it seems to work. In this test program, the first and third print statements output the expected character, and only the second is broken: use Term::ReadLine; binmode(STDOUT, ':encoding(UTF-8)'); print STDOUT ">", chr(0xf3), "<\n"; $Term::ReadLine::Gnu::Attribs{outstream} = \*STDOUT; print STDOUT ">", chr(0xf3), "<\n"; binmode(STDOUT, ':encoding(UTF-8)'); print STDOUT ">", chr(0xf3), "<\n"; Will I have problems if I do it this way? If this works, I don't see why Term::ReadLine::GNU had to remove the encoding layer in the first place. Show quoted text

> > it > > down to the line in the perl source that sets $Attribs{outstream}. I > > looked briefly at the .xs source for _rl_store_iostream, but I can't > > see > > what's causing this. I hope someone who knows perl internals can > > figure > > it out.

> > Here is a comment in Gnu.pm; > --------------------------------------- > # Each variable in the GNU Readline Library is tied to an entry of > # this hash (%Attribs). By accessing the hash entry, you can read > # and/or write the variable in the GNU Readline Library. See the > # package definition of Term::ReadLine::Gnu::Var and following code > # for more details.

I understand that part. I traced the setting of $Attribs{outstream} through the tie into _rl_store_iostream in Gnu.xs. The code there looks like rl_outstream = PerlIO_findFILE(stream); Somehow this breaks my encoding layers, but I don't understand why. Andrew

Wed Aug 18 16:36:12 2010 The RT System itself - Status changed from 'rejected' to 'open'

Thu Aug 19 09:24:07 2010 hiroo.hayashi [...] computer.org - Correspondence added

Subject:	Re: Bug#547231 closed by Salvatore Bonaccorso <salvatore.bonaccorso@gmail.com> ([rt.cpan.org #59832] clobbers binmode layers on filehandles)
Date:	Thu, 19 Aug 2010 22:23:58 +0900
To:	bug-Term-ReadLine-Gnu [...] rt.cpan.org
From:	林宏雄 <hiroo.hayashi [...] computer.org>

Hi, Show quoted text

> rl_outstream = PerlIO_findFILE(stream);

Show quoted text

> Somehow this breaks my encoding layers, but I don't understand why.

I don't, either. As you see, Term::ReadLine::Gnu simply passes the filehandle to the GNU Readline Library directly. Show quoted text

> If Term::ReadLine::Gnu changes the > behavior of the filehandles like this, it should at least be documented.

This may be an undocumented behavior of Perl XS API. -- Hiroo Hayashi

Sun Mar 16 11:26:58 2014 HAYASHI [...] cpan.org - Correspondence added

Hi, Finally I've found a fix for this issue. I found PerlIO::get_layers() function in http://perldoc.perl.org/PerlIO.html. (If I remember correctly, I could not find good documentation of PerlIO when this ticket was opened.) ------------ use Term::ReadLine; my @layers; binmode(STDOUT, ':encoding(utf-8)'); @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; $Term::ReadLine::Gnu::Attribs{outstream} = \*STDOUT; @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; ------------ The code above outputs the following lines. ------------ unix:perlio:encoding(utf-8-strict):utf8 Show quoted text

>ó<

unix:perlio:encoding(utf-8-strict):utf8:stdio Show quoted text

>�<

------------ As you see; rl_outstream = PerlIO_findFILE(stream); adds the stdio layer. By popping the layer, ------------ use Term::ReadLine; my @layers; binmode(STDOUT, ':encoding(utf-8)'); @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; $Term::ReadLine::Gnu::Attribs{outstream} = \*STDOUT; @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; binmode(STDOUT, ":pop"); @layers = PerlIO::get_layers(STDOUT); print join(':', @layers), "\n"; print STDOUT ">", chr(0xf3), "<\n"; ------------ The code above outputs the following lines. ------------ unix:perlio:encoding(utf-8-strict):utf8 Show quoted text

>ó<

unix:perlio:encoding(utf-8-strict):utf8:stdio Show quoted text

>�<

unix:perlio:encoding(utf-8-strict):utf8 Show quoted text

>ó<

------------ Here is the patch. It will be included in Term-ReadLine-Gnu-1.23 soon. --- Gnu.pm (revision 458) +++ Gnu.pm (working copy) @@ -697,7 +698,12 @@ } elsif ($type eq 'F') { return _rl_store_function($value, $id); } elsif ($type eq 'IO') { - return _rl_store_iostream($value, $id); + #my @layers; + my $FH = _rl_store_iostream($value, $id); + #@layers = PerlIO::get_layers($FH); warn "<", join(':', @layers), ">\n"; + binmode($FH, ":pop"); + #@layers = PerlIO::get_layers($FH); warn "<", join(':', @layers), ">\n"; + return $FH; } elsif ($type eq 'K' || $type eq 'LF') { carp "Term::ReadLine::Gnu::Var::STORE: read only variable `$name'\n"; return undef;

Sun Mar 16 11:26:59 2014 HAYASHI [...] cpan.org - Status changed from 'open' to 'patched'

Sun Mar 16 11:26:59 2014 HAYASHI [...] cpan.org - Taken

Wed Mar 19 18:46:04 2014 HAYASHI [...] cpan.org - Correspondence added

The fix is in 1.23.

Wed Mar 19 18:46:05 2014 HAYASHI [...] cpan.org - Status changed from 'patched' to 'resolved'

Wed Mar 19 18:46:05 2014 HAYASHI [...] cpan.org - Fixed in 1.23 added

Sun Mar 23 18:30:32 2014 HAYASHI [...] cpan.org - Correspondence added

The fix in 1.23 did not work with perl 5.8.9 and older. The fix is in 1.24.

Sun Mar 23 18:30:33 2014 HAYASHI [...] cpan.org - Fixed in 1.24 added

Sun Mar 23 18:30:33 2014 HAYASHI [...] cpan.org - Fixed in 1.23 deleted

Bug #59832 for Term-ReadLine-Gnu: clobbers binmode layers on filehandles

Maintainer(s)' notes