Bug #73894 for Term-ReadLine-Gnu: using Term::ReadLine::Gnu mocks binmoded (utf-8) IO

Wed Jan 11 06:05:42 2012 https://launchpad.net/~wanradt - Ticket created

Subject:

using Term::ReadLine::Gnu mocks binmoded (utf-8) IO

Using Term::ReadLine with ::Gnu causes mocked IO. If I comment out creating $term object in below example script i get proper output. Also, if I use Term::ReadLine without Term::ReadLine::Gnu it works fine. --------- #!/usr/bin/env perl use strict; use warnings; use 5.010; use utf8::all; use Term::ReadLine; my $X = <>; chomp $X; say "õ $X"; my $term = new Term::ReadLine ('test', \*STDIN, \*STDOUT); say "õ $X"; $X = <>; chomp $X; say "õ $X"; __END__ got: õ õ õ � � õ � õ ------------- Perl is 5.10.1 and 5.12.4 utf8::all is 0.004 Term::ReadLine is 1.5 Term::ReadLine::Gnu is 1.2 My Kubuntu is set to use UTF-8 aware locale. I rised question also on SO: http://stackoverflow.com/questions/8810852/perl-why-termreadline-spoils- stdin-stdout I hope there is enough details for reproducing, but i could add and test more if you give me some hints.

Thu Jan 12 10:47:56 2012 HAYASHI [...] cpan.org - Correspondence added

Hi, You have to add some lines in your ~/.inputrc. The GNU Readline Library User's Manual http://tiswww.case.edu/php/chet/readline/rluserman.html#SEC9 1.3.3 Sample Init File ... # don't strip characters to 7 bits when reading set input-meta on # allow iso-latin1 characters to be inserted rather # than converted to prefix-meta sequences set convert-meta off # display characters with the eighth bit set directly # rather than as meta-prefixed characters set output-meta on ... I hope this helps you. If not, please provide me the information described in INSTALL file included in the distribution.

Thu Jan 12 10:47:57 2012 The RT System itself - Status changed from 'new' to 'open'

Thu Jan 12 14:49:02 2012 wanradt [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #73894] using Term::ReadLine::Gnu mocks binmoded (utf-8) IO
Date:	Thu, 12 Jan 2012 21:48:53 +0200
To:	bug-Term-ReadLine-Gnu [...] rt.cpan.org
From:	WK <wanradt [...] gmail.com>

Terr! 2012/1/12 Hiroo_HAYASHI via RT <bug-Term-ReadLine-Gnu@rt.cpan.org>: Show quoted text

> You have to add some lines in your ~/.inputrc. > > The GNU Readline Library User's Manual > http://tiswww.case.edu/php/chet/readline/rluserman.html#SEC9 > > 1.3.3 Sample Init File

There is two things involved. First, about 10 years ago i had to set those variables to get international chars to appear right way. Last 5-6 years every distro i tried works fine without explicitly setting them. My conclusion: those vars are set already somewhere else. I tried with .inputrc but this does not change a bit ;( Second, it does not matter, because i have no control over other computers which may use my programs. As without ::Gnu-backend Term::ReadLine works fine and as there is some problem with IO -layers (as pointed in StackOverflow) Show quoted text

> I hope this helps you. If not, please provide me the information > described in INSTALL file included in the distribution.

If you mean Term::ReadLine::Gnu distro, then i use .deb package, which is based on [libterm-readline-gnu-perl_1.20.orig.tar.gz] You can find it here: http://packages.ubuntu.com/oneiric/libterm-readline-gnu-perl I attach also the INSTALL file from this archive. Thank you for your concern! -- Wbr, Kõike hääd, Gunnar

Download INSTALL
application/octet-stream 3.4k

Message body not shown because it is not plain text.

Sat Jan 14 05:08:45 2012 HAYASHI [...] cpan.org - Correspondence added

Hi, Show quoted text

> I tried with .inputrc but this does not change a bit ;(

Thank you for your testing. Show quoted text

> > I hope this helps you. If not, please provide me the information > > described in INSTALL file included in the distribution.

> > If you mean Term::ReadLine::Gnu distro, then i use .deb package, which > is based on [libterm-readline-gnu-perl_1.20.orig.tar.gz] > You can find it here: > http://packages.ubuntu.com/oneiric/libterm-readline-gnu-perl > > I attach also the INSTALL file from this archive.

Sorry, I had to write; If not, please provide me the information described on "Trouble Shooting" section in INSTALL file included in the distribution. ------------------ 1.4 Trouble Shooting If you have any trouble when using or installing this module, please let me (hiroo.hayashi@computer.org) know by E-Mail. It may help other people who have the same problem. I'm sorry that I cannot watch all articles on comp.lang.perl.modules. When you report your trouble, be sure to send me the following information; o result of `perl -V' o compiler you used to compile the GNU Readline Library (libreadline.a). o terminal emulator which you are using o result of `echo $TERM` ------------------ I guess we don't need the information of compiler on this case. Thanks.

Sat Jan 14 10:07:31 2012 wanradt [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #73894] using Term::ReadLine::Gnu mocks binmoded (utf-8) IO
Date:	Sat, 14 Jan 2012 17:07:21 +0200
To:	bug-Term-ReadLine-Gnu [...] rt.cpan.org
From:	WK <wanradt [...] gmail.com>

2012/1/14 Hiroo_HAYASHI via RT <bug-Term-ReadLine-Gnu@rt.cpan.org>: Show quoted text

> o result of `perl -V'

Summary of my perl5 (revision 5 version 12 subversion 4) configuration: Platform: osname=linux, osvers=2.6.24-28-server, archname=i686-linux-gnu-thread-multi-64int uname='linux roseapple 2.6.24-28-server #1 smp wed aug 18 21:17:51 utc 2010 i686 i686 i386 gnulinux ' config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i686-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.12 -Darchlib=/usr/lib/perl/5.12 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.12.4 -Dsitearch=/usr/local/lib/perl/5.12.4 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.12.4 -des' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2 -g', cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.6.1', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /lib/i386-linux-gnu /lib/../lib /usr/lib/i386-linux-gnu /usr/lib/../lib /lib /usr/lib libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt perllibs=-ldl -lm -lpthread -lc -lcrypt libc=, so=so, useshrplib=true, libperl=libperl.so.5.12.4 gnulibc_version='2.13' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API Locally applied patches: DEBPKG:debian/arm_thread_stress_timeout - http://bugs.debian.org/501970 Raise the timeout of ext/threads/shared/t/stress.t to accommodate DEBPKG:debian/cpan_config_path - Set location of CPAN::Config to /etc/perl as /usr may not be DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from DEBPKG:debian/db_file_ver - http://bugs.debian.org/340047 Remove overly restrictive DB_File version check. DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific DEBPKG:debian/enc2xs_inc - http://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories. DEBPKG:debian/errno_ver - http://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running DEBPKG:debian/libperl_embed_doc - http://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking DEBPKG:fixes/respect_umask - Respect umask during installation DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories DEBPKG:debian/extutils_set_libperl_path - EU:MM: Set location of libperl.a to /usr/lib DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor DEBPKG:debian/prefix_changes - Fiddle with *PREFIX and variables written to the makefile DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets. DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor. DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy. DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be DEBPKG:debian/m68k_thread_stress - http://bugs.debian.org/517938 http://bugs.debian.org/495826 Disable some threads tests on m68k for now due to missing TLS. DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian DEBPKG:debian/module_build_man_extensions - http://bugs.debian.org/479460 Adjust Module::Build manual page extensions for the Debian Perl DEBPKG:debian/prune_libs - http://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need. DEBPKG:fixes/net_smtp_docs - [rt.cpan.org #36038] http://bugs.debian.org/100195 Document the Net::SMTP 'Port' option DEBPKG:debian/perlivp - http://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local DEBPKG:debian/disable-zlib-bundling - Disable zlib bundling in Compress::Raw::Zlib DEBPKG:debian/cpanplus_definstalldirs - http://bugs.debian.org/533707 Configure CPANPLUS to use the site directories by default. DEBPKG:debian/cpanplus_config_path - Save local versions of CPANPLUS::Config::System into /etc/perl. DEBPKG:fixes/autodie-flock - http://bugs.debian.org/543731 Allow for flock returning EAGAIN instead of EWOULDBLOCK on DEBPKG:fixes/cpanplus-without-home - http://bugs.debian.org/577011 [rt.cpan.org #52988] Fix CPANPLUS test failures when HOME doesn't exist DEBPKG:debian/deprecate-with-apt - http://bugs.debian.org/580034 Point users to Debian packages of deprecated core modules DEBPKG:fixes/hurd-ccflags - [a190e64] http://bugs.debian.org/587901 [perl #92244] Make hints/gnu.sh append to $ccflags rather than overriding them DEBPKG:debian/squelch-locale-warnings - http://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts DEBPKG:fixes/lc-numeric-docs - [903eb63] [perl #78452] http://bugs.debian.org/379329 LC_NUMERIC documentation fixes DEBPKG:fixes/lc-numeric-sprintf - [b3fd614] [perl #78632] http://bugs.debian.org/601549 Fix sprintf not to ignore LC_NUMERIC with constants DEBPKG:fixes/concat-stack-corruption - [e3393f5] [perl #78674] http://bugs.debian.org/596105 Fix stack pointer corruption in pp_concat() with 'use encoding' DEBPKG:fixes/h2ph-gcc-4.5 - [8d66b3f] http://bugs.debian.org/599933 h2ph fix for gcc 4.5 DEBPKG:fixes/module-build-home-directory - http://bugs.debian.org/624850 [rt.cpan.org #67893] Fix failing tilde test when run under a UID without a passwd entry DEBPKG:debian/patchlevel - http://bugs.debian.org/567489 List packaged patches for 5.12.4-4 in patchlevel.h DEBPKG:fixes/h2ph-multiarch - [e7ec705] http://bugs.debian.org/625808 [perl #90122] Make h2ph correctly search gcc include directories DEBPKG:fixes/odbm-multiarch - [3c998e0] http://bugs.debian.org/625634 [perl #90106] Fix ODBM_File failures on Ubuntu natty [multiarch] DEBPKG:fixes/re.t_sh4_timeout - [7c47ab2] [perl #92226] http://bugs.debian.org/626125 Update lengthen time-out time for t/re/re.t. DEBPKG:fixes/index-tainting - [3b36395] http://bugs.debian.org/291450 [perl #64804] RT 64804: tainting with index() of a constant DEBPKG:fixes/document_makemaker_ccflags - http://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags} DEBPKG:fixes/sys-syslog-socket-timeout-kfreebsd.patch - http://bugs.debian.org/627821 [rt.cpan.org #69997] Use a socket timeout on GNU/kFreeBSD to catch ICMP port unreachable DEBPKG:fixes/hurd-hints - http://bugs.debian.org/636609 Improve general GNU hints, needed for GNU/Hurd. DEBPKG:fixes/encode-heap-overflow - [e46d973] http://bugs.debian.org/637376 Fix decode_xs n-byte heap-overflow security bug in Unicode.xs Built under linux Compiled at Sep 6 2011 07:52:56 %ENV: PERL5LIB="/home/wanradt/perl5/lib/perl5/i686-linux-gnu-thread-multi-64int:/home/wanradt/perl5/lib/perl5" PERL_LOCAL_LIB_ROOT="/home/wanradt/perl5" PERL_MB_OPT="--install_base /home/wanradt/perl5" PERL_MM_OPT="INSTALL_BASE=/home/wanradt/perl5" @INC: /home/wanradt/perl5/lib/perl5/i686-linux-gnu-thread-multi-64int /home/wanradt/perl5/lib/perl5/i686-linux-gnu-thread-multi-64int /home/wanradt/perl5/lib/perl5 /etc/perl /usr/local/lib/perl/5.12.4 /usr/local/share/perl/5.12.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.12 /usr/share/perl/5.12 /usr/local/lib/site_perl . Show quoted text

> o terminal emulator which you are using

mostly Konsole Show quoted text

> o result of `echo $TERM`

xterm I tried also with Perl 5.10.1, with same results. Thank you! -- Wbr, Kõike hääd, Gunnar

Sat Mar 31 23:54:50 2012 HAYASHI [...] cpan.org - Correspondence added

Hi, Sorry for my late reply. Let me confirm what is is the problem. Show quoted text

> I tried also with Perl 5.10.1, with same results.

I have Perl 5.10.1, but it does not have utf8::all. I replaced "utf8::all" with "utf8". ---- $ cat utftest.txt use strict; use warnings; use 5.010; #use utf8::all; use utf8; use Term::ReadLine; my $X = <>; chomp $X; say "õ $X"; my $term = new Term::ReadLine ('test', \*STDIN, \*STDOUT); say "õ $X"; $X = <>; chomp $X; say "õ $X"; $ perl utftest.txt foo � foo � foo bar � bar $ ---------------- õ is replace with �. Next I commented out creating the $term object. ----- $ cat utftest.txt use strict; use warnings; use 5.010; #use utf8::all; use utf8; use Term::ReadLine; my $X = <>; chomp $X; say "õ $X"; #my $term = new Term::ReadLine ('test', \*STDIN, \*STDOUT); say "õ $X"; $X = <>; chomp $X; say "õ $X"; $ perl utftest.txt foo � foo � foo bar � bar $ ---------------- Nothing changed. Next I commented out "use utf8;". --------- $ cat utftest.txt use strict; use warnings; use 5.010; #use utf8::all; #use utf8; use Term::ReadLine; my $X = <>; chomp $X; say "õ $X"; #my $term = new Term::ReadLine ('test', \*STDIN, \*STDOUT); say "õ $X"; $X = <>; chomp $X; say "õ $X"; $ perl utftest.txt foo õ foo õ foo bar õ bar $ ---------------------- õ is printed properly. It helps me if you provide me a script which reproduces your problem on Perl 5.10.1. Regards,

Mon Apr 02 04:31:00 2012 wanradt [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #73894] using Term::ReadLine::Gnu mocks binmoded (utf-8) IO
Date:	Mon, 2 Apr 2012 11:30:49 +0300
To:	bug-Term-ReadLine-Gnu [...] rt.cpan.org
From:	WK <wanradt [...] gmail.com>

2012/4/1 Hiroo_HAYASHI via RT <bug-Term-ReadLine-Gnu@rt.cpan.org>: Show quoted text

> It helps me if you provide me a script which reproduces your problem on > Perl 5.10.1.

"utf8" and "utf8:all" are pretty different beasts: - "utf8" is just saying: in this code is used utf8-encoded chars - "utf8::all" is meant to handle IO properly for UTF-8-based datad and some more From utf8::all docs: utf8 allows you to write your Perl encoded in UTF-8. That means UTF-8 strings, variable names, and regular expressions. utf8::all goes further, and makes @ARGV encoded in UTF-8, and filehandles are opened with UTF-8 encoding turned on by default (including STDIN, STDOUT, STDERR), and charnames are imported so \N{...} sequences can be used to compile Unicode characters based on names. Your first script (with "use utf8") does not handle IO right way (but you input from STDIN just ASCII, so there is no difference). In your second script (without Termline) nothing is changed, because 'use utf8' never step in in IO processing, so there is no problem with it. Your third script (without "use utf8") handles 'õ' as Latin1 encoded 'õ' not UTF-8 encoded 'õ', so everything is happy, until some UTF-8 encoded char comes IN or OUT. From STDIN you should try to read in same char (UTF-8 encoded 'õ' or any other UTF-8 encoded char) and see different behaviour with 'õ' from code and char from STDIN. I used 'õ' and 'ž' in my example script intentionally, because first may appear also in Latin1 too, but other not. So to use them together in one script they should be encoded as UTF-8. Please have a look at Stackoverflow question too, there is script as a whole and great answer as explanation from Daxim, why it happens so. There is some problem with IO layers applied and as far as i understand, this redundant layer is the bug. Seems like one 'stdio' layer is more there. http://stackoverflow.com/questions/8810852 I hope i added some useful bits. TIA, -- Wbr, Kõike hääd, Gunnar

Wed Mar 19 18:53:39 2014 HAYASHI [...] cpan.org - Correspondence added

By having Linux environment I could reproduce your issue. I found this is same issue as #59832. https://rt.cpan.org/Ticket/Display.html?id=59832 I made a fix and it is in 1.23 released last night.

Wed Mar 19 18:53:40 2014 HAYASHI [...] cpan.org - Status changed from 'open' to 'resolved'

Wed Mar 19 18:53:40 2014 HAYASHI [...] cpan.org - Taken

Wed Mar 19 18:53:40 2014 HAYASHI [...] cpan.org - Fixed in 1.23 added

Bug #73894 for Term-ReadLine-Gnu: using Term::ReadLine::Gnu mocks binmoded (utf-8) IO

Maintainer(s)' notes