Bug #113164 for Encode: encode() returns Encode::XS object, dies

Fri Mar 18 21:14:16 2016 nigelgregoire [...] gmail.com - Ticket created

Subject:	encode() returns Encode::XS object, dies
Date:	Fri, 18 Mar 2016 18:13:57 -0700
To:	bug-Encode [...] rt.cpan.org
From:	Nigel Gregoire <nigelgregoire [...] gmail.com>

Hello! On my system encode() sometimes returns an Encode::XS object instead of the resulting string. Here's some sample code: nfg@bill-ubuntu-ng:/home/main/SMS on master *$ cat encode2.pl #!/usr/bin/env perl use strict; use warnings; use Encode; use Data::Dumper; my $str = "You" . chr(8217) . "re doomed!"; binmode STDOUT, ":encoding(utf8)"; binmode STDERR, ":encoding(utf8)"; print $str, "\n"; my $data; my $cb = sub { $data = [ ('?') x 12_500 ]; return ";"; }; my $octets = encode('iso-8859-1', $str, $cb); print Dumper($octets), "\n"; print "And again: ", "\n"; $octets = encode('iso-8859-1', $str, $cb); print Dumper($octets), "\n"; OUTPUT: nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl encode2.pl You’re doomed! $VAR1 = bless( do{\(my $o = '140472427705184')}, 'Encode::XS' ); And again: $VAR1 = 'You;re doomed!'; Segmentation fault (core dumped) I ran into this with some code that uses Text::Unidecode in the CHECK callback. Text::Unidecode can dynamically require packages while running. This is with Encode 2.82. I've reproduced it on Perl 5.8.9, 5.14.2, 5.16.0, 5.20.3, and 5.22.1 , as long as I allocate enough memory. FWIW, Cheers! -- Nigel

Tue Mar 22 04:39:48 2016 DANKOGAI [...] cpan.org - Correspondence added

I could not reproduce your symptom on OS X v10.11.4, Ubuntu 14.04 LTS, and FreeBSD 10.2-RELEASE. Would you tell me more about your environment in details? the output of "perl -V" is pretty helpful. Dan the Maintainer Thereof On Fri Mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote: Show quoted text

> Hello! On my system encode() sometimes returns an Encode::XS object > instead of the resulting string. Here's some sample code: > > nfg@bill-ubuntu-ng:/home/main/SMS on master *$ cat encode2.pl > #!/usr/bin/env perl > use strict; > use warnings; > > use Encode; > use Data::Dumper; > > my $str = "You" . chr(8217) . "re doomed!"; > binmode STDOUT, ":encoding(utf8)"; > binmode STDERR, ":encoding(utf8)"; > print $str, "\n"; > > my $data; > > my $cb = sub { > $data = [ ('?') x 12_500 ]; > return ";"; > }; > > my $octets = encode('iso-8859-1', $str, $cb); > print Dumper($octets), "\n"; > > print "And again: ", "\n"; > $octets = encode('iso-8859-1', $str, $cb); > print Dumper($octets), "\n"; > > OUTPUT: > > nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl encode2.pl > You’re doomed! > $VAR1 = bless( do{\(my $o = '140472427705184')}, 'Encode::XS' ); > > And again: > $VAR1 = 'You;re doomed!'; > > Segmentation fault (core dumped) > > > > I ran into this with some code that uses Text::Unidecode in the CHECK > callback. Text::Unidecode can dynamically require packages while > running. > > This is with Encode 2.82. I've reproduced it on Perl 5.8.9, 5.14.2, > 5.16.0, 5.20.3, and 5.22.1 , as long as I allocate enough memory. > FWIW, > > Cheers! >

Tue Mar 22 04:39:49 2016 The RT System itself - Status changed from 'new' to 'open'

Tue Mar 22 13:29:03 2016 nigelgregoire [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #113164] encode() returns Encode::XS object, dies
Date:	Tue, 22 Mar 2016 10:28:50 -0700
To:	bug-Encode [...] rt.cpan.org
From:	Nigel Gregoire <nigelgregoire [...] gmail.com>

Here you go. FYI, all my testing was done via perlbrew. nfg@bill-ubuntu-ng:/home/main/SMS on master *$ lsb_release -a LSB Version: core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch:core-4.1-amd64:core-4.1-noarch:security-4.0-amd64:security-4.0-noarch:security-4.1-amd64:security-4.1-noarch Distributor ID: Ubuntu Description: Ubuntu 15.10 Release: 15.10 Codename: wily nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl -V Summary of my perl5 (revision 5 version 20 subversion 3) configuration: Platform: osname=linux, osvers=4.2.0-27-generic, archname=x86_64-linux uname='linux bill-ubuntu-ng 4.2.0-27-generic #32-ubuntu smp fri jan 22 04:49:08 utc 2016 x86_64 x86_64 x86_64 gnulinux ' config_args='-de -Dprefix=/home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='5.2.1 20151010', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.21.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.21' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF Locally applied patches: Devel::PatchPerl 1.30 Built under linux Compiled at Mar 18 2016 13:31:29 %ENV: PERL5LIB="/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5/x86_64-linux:/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5" PERLBREW_BASHRC_VERSION="0.43" PERLBREW_HOME="/home/ad/nigelg/.perlbrew" PERLBREW_LIB="dev" PERLBREW_MANPATH="/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/man:/home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/man" PERLBREW_PATH="/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/bin:/home/ad/nigelg/perl5/perlbrew/bin:/home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/bin" PERLBREW_PERL="perl-5.20.3" PERLBREW_ROOT="/home/ad/nigelg/perl5/perlbrew" PERLBREW_VERSION="0.43" PERL_LOCAL_LIB_ROOT="/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev" PERL_MB_OPT="--install_base /home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev" PERL_MM_OPT="INSTALL_BASE=/home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev" @INC: /home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5/x86_64-linux /home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5/x86_64-linux /home/ad/nigelg/.perlbrew/libs/perl-5.20.3@dev/lib/perl5 /home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/lib/site_perl/5.20.3/x86_64-linux /home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/lib/site_perl/5.20.3 /home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/lib/5.20.3/x86_64-linux /home/ad/nigelg/perl5/perlbrew/perls/perl-5.20.3/lib/5.20.3 . On 22 March 2016 at 01:39, Dan Kogai via RT <bug-Encode@rt.cpan.org> wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=113164 > > > I could not reproduce your symptom on OS X v10.11.4, Ubuntu 14.04 LTS, and FreeBSD 10.2-RELEASE. Would you tell me more about your environment in details? > the output of "perl -V" is pretty helpful. > > Dan the Maintainer Thereof > > On Fri Mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote:

>> Hello! On my system encode() sometimes returns an Encode::XS object >> instead of the resulting string. Here's some sample code: >> >> nfg@bill-ubuntu-ng:/home/main/SMS on master *$ cat encode2.pl >> #!/usr/bin/env perl >> use strict; >> use warnings; >> >> use Encode; >> use Data::Dumper; >> >> my $str = "You" . chr(8217) . "re doomed!"; >> binmode STDOUT, ":encoding(utf8)"; >> binmode STDERR, ":encoding(utf8)"; >> print $str, "\n"; >> >> my $data; >> >> my $cb = sub { >> $data = [ ('?') x 12_500 ]; >> return ";"; >> }; >> >> my $octets = encode('iso-8859-1', $str, $cb); >> print Dumper($octets), "\n"; >> >> print "And again: ", "\n"; >> $octets = encode('iso-8859-1', $str, $cb); >> print Dumper($octets), "\n"; >> >> OUTPUT: >> >> nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl encode2.pl >> You’re doomed! >> $VAR1 = bless( do{\(my $o = '140472427705184')}, 'Encode::XS' ); >> >> And again: >> $VAR1 = 'You;re doomed!'; >> >> Segmentation fault (core dumped) >> >> >> >> I ran into this with some code that uses Text::Unidecode in the CHECK >> callback. Text::Unidecode can dynamically require packages while >> running. >> >> This is with Encode 2.82. I've reproduced it on Perl 5.8.9, 5.14.2, >> 5.16.0, 5.20.3, and 5.22.1 , as long as I allocate enough memory. >> FWIW, >> >> Cheers! >>

> > >

-- Nigel http://www.forgreatjustice.ca/

Tue Mar 22 15:44:28 2016 SREZIC [...] cpan.org - Correspondence added

I don't see a segfault either, but valgrind says: ==13517== Invalid write of size 8 ==13517== at 0x609A2B6: XS_Encode__XS_encode (in /bbbike/perl-5.20.1/lib/site_perl/5.20.1/x86_64-linux/auto/Encode/Encode.so) ==13517== by 0x4ABEC6: Perl_pp_entersub (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4A4E32: Perl_runops_standard (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x43FFD3: perl_run (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x422284: main (in /bbbike/perl-5.20.1/bin/perl) ==13517== Address 0x5c98d18 is 8 bytes inside a block of size 1,024 free'd ==13517== at 0x4C28CCE: realloc (vg_replace_malloc.c:632) ==13517== by 0x48AC72: Perl_safesysrealloc (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4A356E: Perl_av_extend_guts (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4D1C19: Perl_stack_grow (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4C4C05: Perl_pp_repeat (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4A4E32: Perl_runops_standard (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4399E2: Perl_call_sv (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x60989BB: do_fallback_cb (in /bbbike/perl-5.20.1/lib/site_perl/5.20.1/x86_64-linux/auto/Encode/Encode.so) ==13517== by 0x609A05A: encode_method (in /bbbike/perl-5.20.1/lib/site_perl/5.20.1/x86_64-linux/auto/Encode/Encode.so) ==13517== by 0x609A2B5: XS_Encode__XS_encode (in /bbbike/perl-5.20.1/lib/site_perl/5.20.1/x86_64-linux/auto/Encode/Encode.so) ==13517== by 0x4ABEC6: Perl_pp_entersub (in /bbbike/perl-5.20.1/bin/perl) ==13517== by 0x4A4E32: Perl_runops_standard (in /bbbike/perl-5.20.1/bin/perl) ==13517== The "Perl_stack_grow" call looks suspicious... Regards, Slaven On 2016-03-22 04:39:48, DANKOGAI wrote: Show quoted text

> I could not reproduce your symptom on OS X v10.11.4, Ubuntu 14.04 LTS, > and FreeBSD 10.2-RELEASE. Would you tell me more about your > environment in details? > the output of "perl -V" is pretty helpful. > > Dan the Maintainer Thereof > > On Fri Mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote:

> > Hello! On my system encode() sometimes returns an Encode::XS object > > instead of the resulting string. Here's some sample code: > > > > nfg@bill-ubuntu-ng:/home/main/SMS on master *$ cat encode2.pl > > #!/usr/bin/env perl > > use strict; > > use warnings; > > > > use Encode; > > use Data::Dumper; > > > > my $str = "You" . chr(8217) . "re doomed!"; > > binmode STDOUT, ":encoding(utf8)"; > > binmode STDERR, ":encoding(utf8)"; > > print $str, "\n"; > > > > my $data; > > > > my $cb = sub { > > $data = [ ('?') x 12_500 ]; > > return ";"; > > }; > > > > my $octets = encode('iso-8859-1', $str, $cb); > > print Dumper($octets), "\n"; > > > > print "And again: ", "\n"; > > $octets = encode('iso-8859-1', $str, $cb); > > print Dumper($octets), "\n"; > > > > OUTPUT: > > > > nfg@bill-ubuntu-ng:/home/main/SMS on master *$ perl encode2.pl > > You’re doomed! > > $VAR1 = bless( do{\(my $o = '140472427705184')}, 'Encode::XS' ); > > > > And again: > > $VAR1 = 'You;re doomed!'; > > > > Segmentation fault (core dumped) > > > > > > > > I ran into this with some code that uses Text::Unidecode in the CHECK > > callback. Text::Unidecode can dynamically require packages while > > running. > > > > This is with Encode 2.82. I've reproduced it on Perl 5.8.9, 5.14.2, > > 5.16.0, 5.20.3, and 5.22.1 , as long as I allocate enough memory. > > FWIW, > > > > Cheers! > >

Fri Apr 22 19:50:40 2016 nigelgregoire [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #113164] encode() returns Encode::XS object, dies
Date:	Fri, 22 Apr 2016 16:50:27 -0700
To:	bug-Encode [...] rt.cpan.org
From:	Nigel Gregoire <nigelgregoire [...] gmail.com>

Hello! I switched to plenv and got a coredump: plenvtest@bill-ubuntu-ng:/home/main/SMS/t/encoding$ gdb ~/.plenv/versions/5.22.1-debug/bin/perl ~/unpacked/CoreDump GNU gdb (Ubuntu 7.10-1ubuntu2) 7.10 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /home/plenvtest/.plenv/versions/5.22.1-debug/bin/perl...done. warning: core file may not match specified executable file. [New LWP 16676] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `perl broken_encode.t'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f4eb03d0267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 55 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) backtrace #0 0x00007f4eb03d0267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55 #1 0x00007f4eb03d1eca in __GI_abort () at abort.c:89 #2 0x00007f4eb0413bf3 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7f4eb052c168 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007f4eb0419767 in malloc_printerr (action=<optimized out>, str=0x7f4eb052829d "corrupted double-linked list", ptr=<optimized Show quoted text

out>) at malloc.c:4965

#4 0x00007f4eb041dc52 in _int_malloc (av=av@entry=0x7f4eb075fc00 <main_arena>, bytes=bytes@entry=1024) at malloc.c:3589 #5 0x00007f4eb0420055 in __libc_calloc (n=<optimized out>, elem_size=<optimized out>) at malloc.c:3213 #6 0x0000000000421db4 in S_new_slab (sz=128) at op.c:181 #7 Perl_Slab_Alloc (sz=9) at op.c:281 #8 0x00000000004266d7 in Perl_newLISTOP (type=192, flags=0, first=0x238eef0, last=0x238ef50) at op.c:4694 #9 0x00000000004674a4 in Perl_yyparse (gramtype=gramtype@entry=258) at perly.y:238 #10 0x00000000004e2e4d in S_doeval (gimme=gimme@entry=2, outside=outside@entry=0x22a77a8, seq=<optimized out>, hh=hh@entry=0x0) at pp_ctl.c:3491 #11 0x00000000004eda02 in Perl_pp_entereval () at pp_ctl.c:4302 #12 0x00000000004ab3b3 in Perl_runops_standard () at run.c:41 #13 0x0000000000442c76 in S_run_body (oldscope=1) at perl.c:2448 #14 perl_run (my_perl=<optimized out>) at perl.c:2371 #15 0x000000000041fcb5 in main (argc=2, argv=0x7ffd725f28b8, env=0x7ffd725f28d0) at perlmain.c:116 Does this help at all? My gdb-fu is weak, but I'll do whatever I can to debug this thing. Cheers! -- Nigel

Fri Sep 30 06:04:30 2016 pali [...] cpan.org - Cc PALI added

Fri Sep 30 06:12:35 2016 pali [...] cpan.org - Correspondence added

On Pia mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote: Show quoted text

> Hello! On my system encode() sometimes returns an Encode::XS object > instead of the resulting string.

Bug in Encode.xs. It modifies perl stack arguments more times between two sequence points. This is undefined behaviour in C. I'm preparing patch for fixing this problem.

Fri Oct 28 08:30:05 2016 pali [...] cpan.org - Correspondence added

On Pia Sep 30 06:12:35 2016, PALI wrote: Show quoted text

> On Pia mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote:

> > Hello! On my system encode() sometimes returns an Encode::XS object > > instead of the resulting string.

> > Bug in Encode.xs. It modifies perl stack arguments more times between > two sequence points. This is undefined behaviour in C. > > I'm preparing patch for fixing this problem.

Pull request with patch: https://github.com/dankogai/p5-encode/pull/72 Fixed in Encode 2.87.

Fri Oct 28 08:30:10 2016 pali [...] cpan.org - Fixed in 2.87 added

Fri Oct 28 13:54:47 2016 nigelgregoire [...] gmail.com - Correspondence added

CC:	pali [...] cpan.org
Subject:	Re: [rt.cpan.org #113164] encode() returns Encode::XS object, dies
Date:	Fri, 28 Oct 2016 10:54:37 -0700
To:	bug-Encode [...] rt.cpan.org
From:	Nigel Gregoire <nigelgregoire [...] gmail.com>

Hey! Thank you so much for looking into this. I just ran my tests on the new Encode and everything passes. :D On 28 October 2016 at 05:30, Pali via RT <bug-Encode@rt.cpan.org> wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=113164 > > > On Pia Sep 30 06:12:35 2016, PALI wrote:

>> On Pia mar 18 21:14:16 2016, nigelgregoire@gmail.com wrote:

>> > Hello! On my system encode() sometimes returns an Encode::XS object >> > instead of the resulting string.

>> >> Bug in Encode.xs. It modifies perl stack arguments more times between >> two sequence points. This is undefined behaviour in C. >> >> I'm preparing patch for fixing this problem.

> > Pull request with patch: https://github.com/dankogai/p5-encode/pull/72 > > Fixed in Encode 2.87.

-- Nigel http://www.forgreatjustice.ca/

Tue Nov 01 09:20:22 2016 DANKOGAI [...] cpan.org - Status changed from 'open' to 'resolved'