Skip Menu |

This queue is for tickets about the Text-Iconv CPAN distribution.

Report information
The Basics
Id: 28850
Status: resolved
Worked: 3 hours (180 min)
Priority: 0/
Queue: Text-Iconv

People
Owner: mxp [...] dynalabs.de
Requestors: aletellier [...] epsiloninteractive.com
Cc: bruno [...] clisp.org
bug-gnu-libiconv [...] gnu.org
AdminCc:

Bug Information
Severity: Critical
Broken in: 1.4
Fixed in: 1.5



Subject: Korean hanging CRITICAL
Text::Iconv hangs while converting from EUC-KR to ISO-2022-KR see attached file, system summary: perl 5.8.8 libiconv 1.10 Solaris 10 x86 perl -V Summary of my perl5 (revision 5 version 8 subversion 8) configuration: Platform: osname=solaris, osvers=2.10, archname=i86pc-solaris-thread-multi uname='sunos dui2 5.10 generic_118855-15 i86pc i386 i86pc ' config_args='-des -Dcc=/usr/local/bin/gcc -Dusethreads' hint=recommended, useposix=true, d_sigaction=define usethreads=define use5005threads=undef useithreads=define usemultiplicity=define useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='/usr/local/bin/gcc', ccflags ='-D_REENTRANT -fno-strict- aliasing -pipe -Wdeclaration-after-statement -I/usr/local/include - D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DPERL_USE_SAFE_PUTENV', optimize='-O', cppflags='-D_REENTRANT -fno-strict-aliasing -pipe -Wdeclaration- after-statement -I/usr/local/include' ccversion='', gccversion='3.4.6', gccosandvers='solaris2.10' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='/usr/local/bin/gcc', ldflags =' -L/usr/local/lib ' libpth=/usr/local/lib /usr/lib /usr/ccs/lib libs=-lsocket -lnsl -lgdbm -ldl -lm -lpthread -lc perllibs=-lsocket -lnsl -ldl -lm -lpthread -lc libc=/lib/libc.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=' ' cccdlflags='-fPIC', lddlflags='-G -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_USE_SAFE_PUTENV USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API Built under solaris Compiled at Dec 29 2006 12:00:29 @INC: /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi /usr/local/lib/perl5/5.8.8 /usr/local/lib/perl5/site_perl/5.8.8/i86pc-solaris-thread-multi /usr/local/lib/perl5/site_perl/5.8.8 /usr/local/lib/perl5/site_perl . /usr/local/bin/iconv --version iconv (GNU libiconv 1.11) Copyright (C) 2000-2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Bruno Haible. uname -a SunOS xxxx 5.10 Generic_118855-15 i86pc i386 i86pc
Subject: iconv_t.pl
use strict; use Text::Iconv (); use FileHandle (); my $euc_kr = 'aldo ¾È³çÇϽÊ'; my $iso_2022_kr = 'aldo $)C>H3gGO=J'; sub text_iconv_t { my ( $data ) = @_; my $iconv = Text::Iconv->new( 'EUC-KR', 'ISO-2022-KR' ); return $iconv->convert( $data ); } sub iconv_t { my ( $data ) = @_; my $file_in = "euc_kr.$$.txt"; my $in_fh = FileHandle->new( $file_in, 'w' ); $in_fh->print( $data ); $in_fh->close(); my $result = `/usr/local/bin/iconv -f EUC-KR -t ISO-2022-KR $file_in`; unlink( $file_in ); return $result; } print 'Iconv ', ( iconv_t( $euc_kr ) eq $iso_2022_kr ) ? 'OK' : 'NOT OK', "\n"; print 'Text::Iconv ', ( text_iconv_t( $euc_kr ) eq $iso_2022_kr ) ? 'OK' : 'NOT OK', "\n";
You used GNU iconv for comparison. Is your Text::Iconv also linked to the GNU libiconv or to Sun's libiconv? At the moment I suspect a problem in the libiconv implementation, since your test program works fine on NetBSD (with the native libiconv).
From: aletellier [...] epsiloninteractive.com
On Sun Aug 19 11:37:23 2007, MPIOTR wrote: Show quoted text
> You used GNU iconv for comparison. Is your Text::Iconv also linked to > the GNU libiconv or to Sun's libiconv? > > At the moment I suspect a problem in the libiconv implementation,
since Show quoted text
> your test program works fine on NetBSD (with the native libiconv).
Thanks so much for the reply, It was compiled with: perl Makefile.PL LIBS='-L/usr/local/lib' INC='-I/usr/local/include' the location of the GNU libiconv, LDD shows: ldd Text/Iconv/Iconv.so libiconv.so.2 => /usr/local/lib/libiconv.so.2 libc.so.1 => /lib/libc.so.1 libgcc_s.so.1 => /usr/local/lib/libgcc_s.so.1 libm.so.2 => /lib/libm.so.2 is there anything else I should look into?
Show quoted text
> Thanks so much for the reply, It was compiled with: > > perl Makefile.PL LIBS='-L/usr/local/lib' INC='-I/usr/local/include' > > the location of the GNU libiconv, LDD shows: > > ldd Text/Iconv/Iconv.so > libiconv.so.2 => /usr/local/lib/libiconv.so.2 > libc.so.1 => /lib/libc.so.1 > libgcc_s.so.1 => /usr/local/lib/libgcc_s.so.1 > libm.so.2 => /lib/libm.so.2
Ok, so this should be the same libiconv. Show quoted text
> is there anything else I should look into?
Hmm. This is very strange: Text::Iconv only calls libiconv functions, so you should always get the same behavior with Text::Iconv as with the iconv(1) utility or other programs using libiconv. Could you try and build Text::Iconv with Sun's libiconv to see whether the problem is related to the library or maybe to something platform-specific? Greetings
From: aletellier [...] epsiloninteractive.com
On Tue Aug 21 06:12:50 2007, MPIOTR wrote: Show quoted text
> > Thanks so much for the reply, It was compiled with: > > > > perl Makefile.PL LIBS='-L/usr/local/lib' INC='-I/usr/local/include' > > > > the location of the GNU libiconv, LDD shows: > > > > ldd Text/Iconv/Iconv.so > > libiconv.so.2 => /usr/local/lib/libiconv.so.2 > > libc.so.1 => /lib/libc.so.1 > > libgcc_s.so.1 => /usr/local/lib/libgcc_s.so.1 > > libm.so.2 => /lib/libm.so.2
> > Ok, so this should be the same libiconv. >
> > is there anything else I should look into?
> > Hmm. This is very strange: Text::Iconv only calls libiconv functions, > so you should always get the same behavior with Text::Iconv as with
the Show quoted text
> iconv(1) utility or other programs using libiconv. > > Could you try and build Text::Iconv with Sun's libiconv to see whether > the problem is related to the library or maybe to something > platform-specific? > > Greetings
Thanks so much for suggestion, Compiled Text::Iconv with Native Sun's iconv, see LDD: ldd Text/Iconv/Iconv.so libc.so.1 => /usr/lib/libc.so.1 libm.so.2 => /usr/lib/libm.so.2 Ran the test program: now using /bin/iconv Not supported EUC-KR to ISO-2022-KR Iconv NOT OK Unsupported conversion: Invalid argument at iconv_t.pl line 10. Text::Iconv NOT OK GNU libiconv supports the above conversion, while Sun's Iconv does not. Please advise.
On Tue Aug 21 20:11:02 2007, ALET wrote: Show quoted text
> > Could you try and build Text::Iconv with Sun's libiconv to see whether > > the problem is related to the library or maybe to something > > platform-specific?
> > Thanks so much for suggestion, > Compiled Text::Iconv with Native Sun's iconv, see LDD: > > ldd Text/Iconv/Iconv.so > libc.so.1 => /usr/lib/libc.so.1 > libm.so.2 => /usr/lib/libm.so.2 > > Ran the test program: now using /bin/iconv > > Not supported EUC-KR to ISO-2022-KR > Iconv NOT OK > Unsupported conversion: Invalid argument at iconv_t.pl line 10. > Text::Iconv NOT OK > > GNU libiconv supports the above conversion, while Sun's Iconv does not.
Yes, I know. The point was to see whether it works at all. I've now been able to try it myself on Solaris and I've seen the same behavior. However, I've also tried Text::Iconv with GNU libiconv on NetBSD, and your test program works fine there. So, it seems that the problem only occurs on Solaris. I'm not yet sure where to start debugging.
CC: bug-gnu-libiconv [...] gnu.org
Subject: Re: [rt.cpan.org #28850] Korean hanging CRITICAL
Date: Wed, 29 Aug 2007 23:53:21 +0200
To: bug-Text-Iconv [...] rt.cpan.org, aletellier [...] epsiloninteractive.com
From: Bruno Haible <bruno [...] clisp.org>
Hello, If you want to show a bug in GNU libiconv, you need to show a C or C++ program that exhibits a behaviour that does not comply with the manual pages. Show quoted text
> > Not supported EUC-KR to ISO-2022-KR > > Iconv NOT OK > > Unsupported conversion: Invalid argument at iconv_t.pl line 10. > > Text::Iconv NOT OK > > > > GNU libiconv supports the above conversion, while Sun's Iconv does not.
This is well-known. On Solaris 10: $ echo abc | /usr/bin/iconv -f EUC-KR -t ISO-2022-KR Not supported EUC-KR to ISO-2022-KR $ echo abc | /usr/bin/iconv -f EUC-KR -t UTF-8 abc $ echo abc | /usr/bin/iconv -f UTF-8 -t ISO-2022-KR abc If you want to perform a conversion from EUC-KR to ISO-2022-KR, you either need an iconv() implementation that supports it directly, such as GNU libc iconv or GNU libiconv, or a utility code that converts via UTF-8 if that is possible. For example you can use the 'striconveh' module from gnulib (it's under LGPL), see http://www.gnu.org/software/gnulib/MODULES.html#module%3Dstriconveh Bruno
CC: <bruno [...] clisp.org>, <bug-gnu-libiconv [...] gnu.org>
Subject: RE: [rt.cpan.org #28850] Korean hanging CRITICAL
Date: Wed, 29 Aug 2007 18:13:24 -0400
To: <bug-Text-Iconv [...] rt.cpan.org>, <aletellier [...] epsiloninteractive.com>
From: "Letellier, Aldo" <ALetellier [...] epsilon.com>
Thanks for your reply, I know about the Sun's Iconv limitations so I work around that by doing a the following: EUC-KR => UTF-8 => ISO-2022-KR Works, but it CORE DUMPS, see gdb log I'm going to do the same trace using GNU libiconv PR_SYSEXIT : Exit from a traced system call Exit from SYS_execve PR_SYSEXIT : Exit from a traced system call Exit from SYS_execve Reading symbols from /usr/lib/ld.so.1...(no debugging symbols found)...done. warning: Temporarily disabling breakpoints for unloaded shared library "/usr/lib/ld.so.1" Reading symbols from /lib/libsocket.so.1...(no debugging symbols found)...done. Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Reading symbols from /lib/libdl.so.1...(no debugging symbols found)...done. Reading symbols from /lib/libm.so.2... (no debugging symbols found)...done. Reading symbols from /lib/libpthread.so.1...warning: Lowest section in /lib/libpthread.so.1 is .dynamic at 00000074 (no debugging symbols found)...done. Reading symbols from /lib/libc.so.1...(no debugging symbols found)...done. Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/IO/IO.so... (no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Fcntl/Fcntl.s o...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Storable/Stor able.so...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/POSIX/POSIX.s o...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Encode/Encode .so... (no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Encode/Unicod e/Unicode.so...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/List/Util/Uti l.so...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Encode/JP/JP. so...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/site_perl/5.8.8/i86pc-solaris-thread-multi/auto/Tex t/Iconv/Iconv.so... (no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/File/Glob/Glo b.so...(no debugging symbols found)...done. From: US-ASCII To: UTF-8 Reading symbols from /usr/lib/iconv/646%UTF-8.so...(no debugging symbols found)...done. Reading symbols from /usr/lib/iconv/646%UTF-8.so...(no debugging symbols found)...done. From: EUC-KR To: ISO-2022-KR Reading symbols from /usr/lib/iconv/EUC-KR%UTF-8.so... (no debugging symbols found)...done. Reading symbols from /usr/lib/iconv/UTF-8%ISO-2022-KR.so...(no debugging symbols found)...done. Reading symbols from /usr/lib/iconv/EUC-KR%UTF-8.so...(no debugging symbols found)...done. Reading symbols from /usr/lib/iconv/UTF-8%ISO-2022-KR.so...(no debugging symbols found)...done. From: ISO-2022-KR To: EUC-KR PR_FAULTED : Incurred a traced hardware fault FLTBOUNDS: Memory bounds violation Program received signal SIGSEGV, Segmentation fault. 0xfedc33e7 in realfree () from /lib/libc.so.1 Show quoted text
-----Original Message----- From: Michael Piotrowski via RT [mailto:bug-Text-Iconv@rt.cpan.org] Sent: Wednesday, August 29, 2007 2:29 PM To: aletellier@epsiloninteractive.com Cc: bruno@clisp.org; bug-gnu-libiconv@gnu.org Subject: [rt.cpan.org #28850] Korean hanging CRITICAL <URL: http://rt.cpan.org/Ticket/Display.html?id=28850 > On Tue Aug 21 20:11:02 2007, ALET wrote:
> > Could you try and build Text::Iconv with Sun's libiconv to see
whether
> > the problem is related to the library or maybe to something > > platform-specific?
> > Thanks so much for suggestion, > Compiled Text::Iconv with Native Sun's iconv, see LDD: > > ldd Text/Iconv/Iconv.so > libc.so.1 => /usr/lib/libc.so.1 > libm.so.2 => /usr/lib/libm.so.2 > > Ran the test program: now using /bin/iconv > > Not supported EUC-KR to ISO-2022-KR > Iconv NOT OK > Unsupported conversion: Invalid argument at iconv_t.pl line 10. > Text::Iconv NOT OK > > GNU libiconv supports the above conversion, while Sun's Iconv does
not. Yes, I know. The point was to see whether it works at all. I've now been able to try it myself on Solaris and I've seen the same behavior. However, I've also tried Text::Iconv with GNU libiconv on NetBSD, and your test program works fine there. So, it seems that the problem only occurs on Solaris. I'm not yet sure where to start debugging.
CC: <bug-gnu-libiconv [...] gnu.org>
Subject: RE: [rt.cpan.org #28850] Korean hanging CRITICAL
Date: Wed, 29 Aug 2007 18:43:00 -0400
To: <bug-Text-Iconv [...] rt.cpan.org>, <aletellier [...] epsiloninteractive.com>
From: "Letellier, Aldo" <ALetellier [...] epsilon.com>
Ok found something looks like Text::Iconv hangs at unicode_loop_convert () from /usr/local/lib/libiconv.so.2 I'm now using GNU libiconv 1.11 Let me try compiling libiconv with debug optimizer GNU gdb 6.6 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-pc-solaris2.10"... (no debugging symbols found) (gdb) set verbose 1 (gdb) run korean_t.pl Starting program: /usr/bin/perl korean_t.pl PR_SYSEXIT : Exit from a traced system call Exit from SYS_execve PR_SYSEXIT : Exit from a traced system call Exit from SYS_execve Reading symbols from /usr/lib/ld.so.1...(no debugging symbols found)...done. warning: Temporarily disabling breakpoints for unloaded shared library "/usr/lib/ld.so.1" Reading symbols from /lib/libsocket.so.1...(no debugging symbols found)...done. Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Reading symbols from /lib/libdl.so.1...(no debugging symbols found)...done. Reading symbols from /lib/libm.so.2... (no debugging symbols found)...done. Reading symbols from /lib/libpthread.so.1...warning: Lowest section in /lib/libpthread.so.1 is .dynamic at 00000074 (no debugging symbols found)...done. Reading symbols from /lib/libc.so.1...(no debugging symbols found)...done. Reading symbols from /lib/ld.so.1...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/IO/IO.so... (no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Fcntl/Fcntl.s o...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Storable/Stor able.so...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/POSIX/POSIX.s o...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Encode/Encode .so... (no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Encode/Unicod e/Unicode.so...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/List/Util/Uti l.so...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/Encode/JP/JP. so...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/perl5/site_perl/5.8.8/i86pc-solaris-thread-multi/auto/Tex t/Iconv/Iconv.so... (no debugging symbols found)...done. Reading symbols from /usr/local/lib/libiconv.so.2...(no debugging symbols found)...done. Reading symbols from /usr/local/lib/libgcc_s.so.1...done. Reading symbols from /usr/local/lib/perl5/5.8.8/i86pc-solaris-thread-multi/auto/File/Glob/Glo b.so...done. 123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx yz{|} From: US-ASCII To: UTF-8 Iconv OK From: EUC-KR To: ISO-2022-KR Iconv OK ^CPR_SIGNALLED : Receipt of a traced signal SIGINT: Interrupt (rubout) Program received signal SIGINT, Interrupt. 0xfe9f26b0 in unicode_loop_convert () from /usr/local/lib/libiconv.so.2 (gdb) Show quoted text
-----Original Message----- From: bruno@clisp.org via RT [mailto:bug-Text-Iconv@rt.cpan.org] Sent: Wednesday, August 29, 2007 5:54 PM To: aletellier@epsiloninteractive.com Cc: bug-gnu-libiconv@gnu.org Subject: Re: [rt.cpan.org #28850] Korean hanging CRITICAL <URL: http://rt.cpan.org/Ticket/Display.html?id=28850 > Hello, If you want to show a bug in GNU libiconv, you need to show a C or C++ program that exhibits a behaviour that does not comply with the manual pages.
> > Not supported EUC-KR to ISO-2022-KR > > Iconv NOT OK > > Unsupported conversion: Invalid argument at iconv_t.pl line 10. > > Text::Iconv NOT OK > > > > GNU libiconv supports the above conversion, while Sun's Iconv does
not. This is well-known. On Solaris 10: $ echo abc | /usr/bin/iconv -f EUC-KR -t ISO-2022-KR Not supported EUC-KR to ISO-2022-KR $ echo abc | /usr/bin/iconv -f EUC-KR -t UTF-8 abc $ echo abc | /usr/bin/iconv -f UTF-8 -t ISO-2022-KR abc If you want to perform a conversion from EUC-KR to ISO-2022-KR, you either need an iconv() implementation that supports it directly, such as GNU libc iconv or GNU libiconv, or a utility code that converts via UTF-8 if that is possible. For example you can use the 'striconveh' module from gnulib (it's under LGPL), see http://www.gnu.org/software/gnulib/MODULES.html#module%3Dstriconveh Bruno
CC: bug-gnu-libiconv [...] gnu.org
Subject: Re: [bug-gnu-libiconv] RE: [rt.cpan.org #28850] Korean hanging CRITICAL
Date: Thu, 30 Aug 2007 01:15:29 +0200
To: "Letellier, Aldo" <ALetellier [...] epsilon.com>, <bug-Text-Iconv [...] rt.cpan.org>, <aletellier [...] epsiloninteractive.com>
From: Bruno Haible <bruno [...] clisp.org>
Show quoted text
> ^CPR_SIGNALLED : Receipt of a traced signal SIGINT: Interrupt (rubout) > > > Program received signal SIGINT, Interrupt. > 0xfe9f26b0 in unicode_loop_convert () from /usr/local/lib/libiconv.so.2
If you want to prove convincingly that there is a bug in libiconv, the best way is to look at the arguments being passed to the iconv() function from libiconv, and write a small C program with the same inputs and that fails or loops. Bruno
I've found the problem, and it's in Text::Iconv. The problem is that the output buffer is too small. It seems that I've set it to a very small size (5 bytes) for testing the reallocation and inadvertently left that in the released version. I don't know why it only seems to occur on Solaris, but in any case this is a bug and I'll make a new release later today. You may want to try whether changing the line outbytesleft = 5; /* 2 * inbytesleft; */ in Iconv.xs to outbytesleft = 2 * inbytesleft; fixes the problem for you.
Fixed in Text::Iconv 1.5.