Skip Menu |

This queue is for tickets about the Term-ReadLine-Gnu CPAN distribution.

Maintainer(s)' notes

When you report a bug, please provide the following information;

- output of
	perl -V
	perl Makefile.PL verbose
	make test TEST_VERBOSE=1
	perl -Mblib t/00checkver.t
	echo $TERM
- terminal emulator which you are using
- compiler which is used to compile the GNU Readline Library (libreadline.a) if you can know.
Read INSTALL in the distribution for more details.

Report information
The Basics
Id: 114231
Status: resolved
Priority: 0/
Queue: Term-ReadLine-Gnu

People
Owner: HAYASHI [...] cpan.org
Requestors: ambs [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: 1.32



Subject: utf 8 is still tricky
Hello using the latest version on CPAN, and doing binmode to the OUT filehandle, UTF8 is correctly printed to the screen. No need to change locale or whatsoever. nevertheless, input is not working yet. It no matters if I binmode or not the IN filehandle, and if I change locale or not. My current solution is to call Encode::decode on the string returned by readline, as it is a octet sequence, and not an utf8 string. Best Alberto
Hi, The recent versions include UTF8 tests. They pass on many environments. http://www.cpantesters.org/distro/T/Term-ReadLine-Gnu.html (See https://rt.cpan.org/Ticket/Display.html?id=114235#txn-1624538 for more details.) Give me more information. - your script which fails - output of perl -Mblib t/utf8_text.t and - output of perl -V perl Makefile.PL verbose make test TEST_VERBOSE=1 perl -Mblib t/00checkver.t echo $TERM - terminal emulator which you are using - compiler which is used to compile the GNU Readline Library (libreadline.a) if you can know. On Sat, 7 May 2016 13:54:45 GMT, AMBS wrote: Show quoted text
> Hello > > using the latest version on CPAN, and doing binmode to the OUT > filehandle, UTF8 is correctly printed to the screen. No need to change > locale or whatsoever. > > nevertheless, input is not working yet. It no matters if I binmode or > not the IN filehandle, and if I change locale or not. My current > solution is to call Encode::decode on the string returned by readline, > as it is a octet sequence, and not an utf8 string. > > Best > Alberto
Subject: Re: [rt.cpan.org #114231] utf 8 is still tricky
Date: Sun, 8 May 2016 16:36:00 +0100
To: bug-Term-ReadLine-Gnu [...] rt.cpan.org, ambs [...] cpan.org
From: Alberto Simoes <ambs [...] perl-hackers.net>
Hi, Sure, will try to make a failing test during this week. Cheers On 08/05/16 16:25, Hiroo_HAYASHI via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=114231 > > > Hi, > > The recent versions include UTF8 tests. > They pass on many environments. > http://www.cpantesters.org/distro/T/Term-ReadLine-Gnu.html > (See https://rt.cpan.org/Ticket/Display.html?id=114235#txn-1624538 for more details.) > > Give me more information. > > - your script which fails > - output of perl -Mblib t/utf8_text.t > > and > > - output of > perl -V > perl Makefile.PL verbose > make test TEST_VERBOSE=1 > perl -Mblib t/00checkver.t > echo $TERM > - terminal emulator which you are using > - compiler which is used to compile the GNU Readline Library (libreadline.a) if you can know. > > > On Sat, 7 May 2016 13:54:45 GMT, AMBS wrote:
>> Hello >> >> using the latest version on CPAN, and doing binmode to the OUT >> filehandle, UTF8 is correctly printed to the screen. No need to change >> locale or whatsoever. >> >> nevertheless, input is not working yet. It no matters if I binmode or >> not the IN filehandle, and if I change locale or not. My current >> solution is to call Encode::decode on the string returned by readline, >> as it is a octet sequence, and not an utf8 string. >> >> Best >> Alberto
>
Hi Sorry for taking too long, but work and life didn't help. Also, sorry for the long e-mail :) I noticed that in tests you are using locale "en_US.utf8" My mac fails to set that locale and therefore skips your utf8 tests. Trying to understand why, I noticed that if I change the locale to "en_US.UTF-8" (note the uppercase and hyphen), the Mac starts running those tests (and passes them!). To have different locales given the OS is not cute. So I went out to my linux machine, and tried changing your tests to "en_US.UTF-8" and they work. After that I googled in order to understand if there is a standard. As far as I could understand, there is a standard up to the dot, but the encoding itself is kind of savage. Some defend that the uppercase versions is the correct. Some others point to those that their documentation isn't consistent and use the uppercase and lowercase versions mixed up. So, ignoring what is to be supposed the standard, the truth is that Linux as aliases that support the uppercase version. So, I would suggest you to change the tests. Basically, s/en_US.utf8/en_US.UTF-8/g in your tests. I am not yet sure if this was my problem in my script. But now that I found out it, I will test again, and will let you know on what I find. Best regards, Alberto On Sun May 08 11:36:24 2016, ambs@perl-hackers.net wrote: Show quoted text
> > Hi, > Sure, will try to make a failing test during this week. > Cheers > > On 08/05/16 16:25, Hiroo_HAYASHI via RT wrote:
> > <URL: https://rt.cpan.org/Ticket/Display.html?id=114231 > > > > > Hi, > > > > The recent versions include UTF8 tests. > > They pass on many environments. > > http://www.cpantesters.org/distro/T/Term-ReadLine-Gnu.html > > (See https://rt.cpan.org/Ticket/Display.html?id=114235#txn-1624538 > > for more details.) > > > > Give me more information. > > > > - your script which fails > > - output of perl -Mblib t/utf8_text.t > > > > and > > > > - output of > > perl -V > > perl Makefile.PL verbose > > make test TEST_VERBOSE=1 > > perl -Mblib t/00checkver.t > > echo $TERM > > - terminal emulator which you are using > > - compiler which is used to compile the GNU Readline Library > > (libreadline.a) if you can know. > > > > > > On Sat, 7 May 2016 13:54:45 GMT, AMBS wrote:
> >> Hello > >> > >> using the latest version on CPAN, and doing binmode to the OUT > >> filehandle, UTF8 is correctly printed to the screen. No need to > >> change > >> locale or whatsoever. > >> > >> nevertheless, input is not working yet. It no matters if I binmode > >> or > >> not the IN filehandle, and if I change locale or not. My current > >> solution is to call Encode::decode on the string returned by > >> readline, > >> as it is a octet sequence, and not an utf8 string. > >> > >> Best > >> Alberto
> >
Hi, OK, More information 1) a test script in attach. My test word is "coração" Show quoted text
> > > - output of perl -Mblib t/utf8_text.t
After that utf8 => UTF-8 change: ambs@stravinski Term-ReadLine-Gnu-1.31 $ perl -Mblib t/utf8_text.t 1..4 # Running under perl version 5.024000 for darwin # Current time local: Mon May 23 14:44:29 2016 # Current time GMT: Mon May 23 13:44:29 2016 # Using Test.pm version 1.28 # I'm testing Term::ReadLine::Gnu version 1.31 ok 1 #i: unix:perlio:encoding(utf8):utf8 #o: unix:perlio:encoding(utf8):utf8 ok 2 #i: unix:perlio:encoding(utf8):utf8 #o: unix:perlio:encoding(utf8):utf8 Show quoted text
漢字> 漢字1
漢字1 $VAR1 = "\x{6f22}\x{5b57}1"; $VAR2 = "\x{6f22}\x{5b57}1"; ok 3 ok 4 ambs@stravinski Term-ReadLine-Gnu-1.31 $ Show quoted text
> > > - output of > > > perl -V > > > perl Makefile.PL verbose > > > make test TEST_VERBOSE=1
all in attach Show quoted text
> > > perl -Mblib t/00checkver.t
1..4 # Running under perl version 5.024000 for darwin # Current time local: Mon May 23 14:47:14 2016 # Current time GMT: Mon May 23 13:47:14 2016 # Using Test.pm version 1.28 # I'm testing Term::ReadLine::Gnu version 1.31 ok 1 ok 2 ok 3 # OS: darwin # Perl version: 5.024000 # GNU Readline Library version: 6.3, 0x603 # $TERM=xterm-256color ok 4 Show quoted text
> > > echo $TERM
xterm-256color Show quoted text
> > > - terminal emulator which you are using
iterm2 Show quoted text
> > > - compiler which is used to compile the GNU Readline Library > > > (libreadline.a) if you can know.
Apple LLVM version 7.0.0 (clang-700.1.75) Target: x86_64-apple-darwin15.4.0 Thread model: posix Cheers Alberto
Subject: MakefileVerbose
Download MakefileVerbose
application/octet-stream 2k

Message body not shown because it is not plain text.

Subject: makeTest
Download makeTest
application/octet-stream 6.7k

Message body not shown because it is not plain text.

Subject: perlV.txt
Summary of my perl5 (revision 5 version 24 subversion 0) configuration: Platform: osname=darwin, osvers=15.4.0, archname=darwin-2level uname='darwin stravinski.local 15.4.0 darwin kernel version 15.4.0: fri feb 26 22:08:05 pst 2016; root:xnu-3248.40.184~3release_x86_64 x86_64 ' config_args='-Dprefix=/opt/perl' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.11 -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -DPERL_USE_SAFE_PUTENV', optimize='-O3', cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.11 -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include' ccversion='', gccversion='4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.1.75)', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -mmacosx-version-min=10.11 -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /Volumes/HDD/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/7.0.0/lib /Volumes/HDD/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib /usr/lib libs=-lpthread -lgdbm -ldbm -ldb -ldl -lm -lutil -lc perllibs=-lpthread -ldl -lm -lutil -lc libc=, so=dylib, useshrplib=false, libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags=' -mmacosx-version-min=10.11 -bundle -undefined dynamic_lookup -L/usr/local/lib -fstack-protector-strong' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP PERL_PRESERVE_IVUV PERL_USE_SAFE_PUTENV USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF Built under darwin Compiled at May 9 2016 15:04:53 %ENV: PERL_AUTOINSTALL_PREFER_CPAN="1" PERL_CPANM_OPT="-S" @INC: /opt/perl/lib/site_perl/5.24.0/darwin-2level /opt/perl/lib/site_perl/5.24.0 /opt/perl/lib/5.24.0/darwin-2level /opt/perl/lib/5.24.0 .
Subject: stest.pl
# ABSTRACT: Simple shell to interact with PULO use strict; use warnings; use URI::Encode 'uri_encode'; use Term::ReadLine; use utf8; use open ':encoding(utf8)'; use open ':std'; use POSIX qw(locale_h); use locale; BEGIN { $ENV{LC_ALL} = "en_US.UTF-8"; my $old_locale = setlocale(LC_ALL, 'en_US.UTF-8'); } my $term = Term::ReadLine->new("pulo-shell"); my $line = $term->readline("> "); print {$term->OUT} "IS UTF8\n" if utf8::is_utf8($line); print {$term->OUT} $line, "\n"; print {$term->OUT} "Encoded: ", uri_encode($line), "\n"; print {$term->OUT} "Should be: ", uri_encode("coração"), "\n";
And forgot to paste the output of my script: ambs@stravinski tmp $ perl stest.pl Show quoted text
> coração
coração Encoded: cora%C3%83%C2%A7%C3%83%C2%A3o Should be: cora%C3%A7%C3%A3o Notice that it appears correctly even in Print. But the line checking if it is UTF8 doesn't show (so, probably, it is not utf8) Notice how URL encoding is double encoding; And what should be if it was encoding utf8. Thanks
Hi Alberto, Thank you for your report. It means that this issue is a duplication of #114235. https://rt.cpan.org/Public/Bug/Display.html?id=114235#txn-1626868 Fix is already in the SVN. I learned that en_US.utf8 is valid only for Linux and ports from it (Cygwin, etc.). Before releasing my module, I searched which was better, en_US.UTF-8 or en_US.utf8, but I could not find an answer. Thank you for your help again.
On Mon May 23 10:34:32 2016, HAYASHI wrote: Show quoted text
> Hi Alberto, > > Thank you for your report. > > It means that this issue is a duplication of #114235. > https://rt.cpan.org/Public/Bug/Display.html?id=114235#txn-1626868 > > Fix is already in the SVN. >
Note that that was not the only problem. Please check and test the stest script above. I tried it with your svn version and it still doesn't work as expected. Not sure if still a bug on trg or in my script. Thanks
RT-Send-CC: ambs [...] cpan.org
Hi, You have to replace the line Show quoted text
> my $term = Term::ReadLine->new("pulo-shell");
with Show quoted text
> my $term = Term::ReadLine->new("pulo-shell", \*STDIN, \*STDOUT);
The open pragma ("use open ':std', ':encoding(utf8)';") is lexically scoped. Term::ReadLine->new() opens /dev/tty for input and output filehandles when filehandles is not specified on its argument. Open /dev/tty is not affected by the open pragma. $ PERL_RL=Gnu perl -Mblib use_open_test.pl unix:perlio:stdio unix:perlio:encoding(utf8):utf8 Only the second new adds the utf8 layer. The standard Term::ReadLine behaves similarly. $ PERL_RL=Stub perl -Mblib use_open_test.pl unix:perlio unix:perlio:encoding(utf8):utf8 A good way to support UTF-8 was proposed on https://rt.cpan.org/Ticket/Display.html?id=104239. I am glad to support it, if the standard Term::ReadLine include the feature. On Mon, 23 May 2016 14:48:22 GMT, AMBS wrote: Show quoted text
> On Mon May 23 10:34:32 2016, HAYASHI wrote:
> > Hi Alberto, > > > > Thank you for your report. > > > > It means that this issue is a duplication of #114235. > > https://rt.cpan.org/Public/Bug/Display.html?id=114235#txn-1626868 > > > > Fix is already in the SVN. > >
> > Note that that was not the only problem. Please check and test the > stest script above. I tried it with your svn version and it still > doesn't work as expected. Not sure if still a bug on trg or in my > script. > > Thanks
Subject: use_open_test.pl
#!/usr/bin/perl use strict; use warnings; use open ':std', ':encoding(utf8)'; use Term::ReadLine; my $t; $t = new Term::ReadLine 'foo'; print_layers($t->IN); $t = new Term::ReadLine 'test', \*STDIN, \*STDOUT; print_layers($t->IN); exit 0; sub print_layers { my $fh = shift; my @layers = PerlIO::get_layers($fh); print join(':', @layers), "\n"; }
CC: ambs [...] cpan.org
Subject: Re: [rt.cpan.org #114231] utf 8 is still tricky
Date: Tue, 24 May 2016 09:19:29 +0100
To: bug-Term-ReadLine-Gnu [...] rt.cpan.org
From: Alberto Simoes <ambs [...] perl-hackers.net>
On 24/05/16 07:14, Hiroo_HAYASHI via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=114231 > > > Hi, > > You have to replace the line
>> my $term = Term::ReadLine->new("pulo-shell");
> > with
>> my $term = Term::ReadLine->new("pulo-shell", \*STDIN, \*STDOUT);
>
Ah, that explains it. Will try and probably suggest a patch to document it (while T::RL doesn't have an utf-8 aware API -- I recall reading something on that on p5p). Thank you
On Tue May 24 04:19:50 2016, ambs@perl-hackers.net wrote: Show quoted text
> On 24/05/16 07:14, Hiroo_HAYASHI via RT wrote:
> > <URL: https://rt.cpan.org/Ticket/Display.html?id=114231 > > > > > Hi, > > > > You have to replace the line
> >> my $term = Term::ReadLine->new("pulo-shell");
> > > > with
> >> my $term = Term::ReadLine->new("pulo-shell", \*STDIN, \*STDOUT);
> >
> > Ah, that explains it. > > Will try and probably suggest a patch to document it (while T::RL > doesn't have an utf-8 aware API -- I recall reading something on that on > p5p). > > Thank you >
Just to let you know that passing the references to STDIN-STDOUT worked like a charm. Thanks and sorry for the delay!
Show quoted text
> Just to let you know that passing the references to STDIN-STDOUT > worked like a charm. > > Thanks and sorry for the delay!
You are welcome. I released 1.32 last night. I added enableUTF8 method. https://rt.cpan.org/Ticket/Display.html?id=104239#txn-1635895