Bug #84493 for Encode: Sometimes hang under FreeBSD when encoding layer used and encodings are incorrect.

Sun Apr 07 16:42:59 2013 victor [...] vsespb.ru - Ticket created

Subject:

Sometimes hang under FreeBSD when encoding layer used and encodings are incorrect.

perl -MEncode -e 'print Encode->VERSION' 2.42_01 The following commands hang with 50% CPU on FreeBSD 9.1 (This is perl 5, version 14, subversion 2 (v5.14.2) built for amd64-freebsd ) perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}\x{43a}")' perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}")' and the following work fine: perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}")' perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess encode("utf-8", "\x{440}\x{43a}")' $ locale LANG= LC_CTYPE="C" LC_COLLATE="C" LC_TIME="C" LC_NUMERIC="C" LC_MONETARY="C" LC_MESSAGES="C" LC_ALL= also, cannot reproduce under linux possibly related to #84282

Sun Apr 07 18:44:05 2013 victor [...] vsespb.ru - Correspondence added

UPD: perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; croak "\320\220\320\262\321\202\320\276\320\277\320\260\321\200\320\272"' hands too. If I replace 'croak' with die, it works. If I remove "encoding" layer it works too. Also all this PoC works without bugs on FreeBSD 8 with KOI8-R locale. On Mon Apr 08 00:42:59 2013, vsespb wrote: Show quoted text

> perl -MEncode -e 'print Encode->VERSION' > 2.42_01 > > > The following commands hang with 50% CPU on > FreeBSD 9.1 (This is perl 5, version 14, subversion 2 (v5.14.2) built > for amd64-freebsd > ) > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", > "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}\x{43a}")' > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}")' > > > and the following work fine: > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}")' > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", "\x{440}\x{43a}")' > > > $ locale > LANG= > LC_CTYPE="C" > LC_COLLATE="C" > LC_TIME="C" > LC_NUMERIC="C" > LC_MONETARY="C" > LC_MESSAGES="C" > LC_ALL= > > also, cannot reproduce under linux > > possibly related to #84282 >

Sun Apr 07 18:44:06 2013 victor [...] vsespb.ru - Status changed from 'new' to 'open'

Mon Apr 08 09:38:20 2013 DANKOGAI [...] cpan.org - Correspondence added

You should not use PerlIO on STDERR because PerlIO modules may carp on STDERR, resulting deep recursion and such. Dan the Maintainer Thereof On Sun Apr 07 16:42:59 2013, vsespb wrote: Show quoted text

> perl -MEncode -e 'print Encode->VERSION' > 2.42_01 > > > The following commands hang with 50% CPU on > FreeBSD 9.1 (This is perl 5, version 14, subversion 2 (v5.14.2) built > for amd64-freebsd > ) > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", > "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}\x{43a}")' > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}")' > > > and the following work fine: > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}")' > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", "\x{440}\x{43a}")' > > > $ locale > LANG= > LC_CTYPE="C" > LC_COLLATE="C" > LC_TIME="C" > LC_NUMERIC="C" > LC_MONETARY="C" > LC_MESSAGES="C" > LC_ALL= > > also, cannot reproduce under linux > > possibly related to #84282 >

Mon Apr 08 09:38:26 2013 DANKOGAI [...] cpan.org - Status changed from 'open' to 'resolved'

Mon Apr 08 09:50:56 2013 victor [...] vsespb.ru - Correspondence added

Hm. Is it documented ? How then stacktrace (croak/carp/die) with non-ASCII data supposed to work ? There are some publications http://www.perl.com/pub/2012/04/perlunicook-decode-standard-filehandles-as-utf-8.html which advice to use PerlIO on stderr. Also there is a pragma 'use open :std' in the CORE. On Mon Apr 08 17:38:20 2013, DANKOGAI wrote: Show quoted text

> You should not use PerlIO on STDERR because PerlIO modules may carp on > STDERR, resulting deep recursion and such. > > Dan the Maintainer Thereof > > On Sun Apr 07 16:42:59 2013, vsespb wrote:

> > perl -MEncode -e 'print Encode->VERSION' > > 2.42_01 > > > > > > The following commands hang with 50% CPU on > > FreeBSD 9.1 (This is perl 5, version 14, subversion 2 (v5.14.2)

> built

> > for amd64-freebsd > > ) > > > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)";

> confess

> > encode("utf-8", > > "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}\x{43a}")' > > > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)";

> confess

> > encode("utf-8",

> "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}")'

> > > > > > and the following work fine: > > > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)";

> confess

> > encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}")' > > > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)";

> confess

> > encode("utf-8", "\x{440}\x{43a}")' > > > > > > $ locale > > LANG= > > LC_CTYPE="C" > > LC_COLLATE="C" > > LC_TIME="C" > > LC_NUMERIC="C" > > LC_MONETARY="C" > > LC_MESSAGES="C" > > LC_ALL= > > > > also, cannot reproduce under linux > > > > possibly related to #84282 > >

> >

Mon Apr 08 09:50:57 2013 The RT System itself - Status changed from 'resolved' to 'open'

Mon Apr 08 10:03:42 2013 DANKOGAI [...] cpan.org - Correspondence added

PerlIO handlers like ':utf8' are okay because they do not modify data. Some are not since they do modify data (and more importantly, its length) ':encoding()' is exactly that. I agree we should document that but it should be more documented in PerlIO part... Dan On Mon Apr 08 09:50:56 2013, vsespb wrote: Show quoted text

> Hm. > > Is it documented ? > > How then stacktrace (croak/carp/die) with non-ASCII data supposed to > work ? > > There are some publications > http://www.perl.com/pub/2012/04/perlunicook-decode-standard- > filehandles-as-utf-8.html which advice to use PerlIO on stderr. > > Also there is a pragma 'use open :std' in the CORE.

Mon Apr 08 10:03:43 2013 DANKOGAI [...] cpan.org - Status changed from 'open' to 'resolved'

Mon Apr 08 10:11:28 2013 victor [...] vsespb.ru - Correspondence added

Ok, clear. Well, :utf8 can modify data sometimes, I believe(when binary data treated as Latin-1) $perl -e 'binmode STDOUT, ":utf8"; print "\xDC"'|wc 0 1 2 $perl -e ' print "\xDC"'|wc 0 0 1 OK, I will submit perlbug for PerlIO then. On Mon Apr 08 18:03:42 2013, DANKOGAI wrote: Show quoted text

> PerlIO handlers like ':utf8' are okay because they do not modify data. > Some are not since they do modify data (and more importantly, its > length) ':encoding()' is exactly that. > > I agree we should document that but it should be more documented in > PerlIO part... > > Dan > > On Mon Apr 08 09:50:56 2013, vsespb wrote:

> > Hm. > > > > Is it documented ? > > > > How then stacktrace (croak/carp/die) with non-ASCII data supposed to > > work ? > > > > There are some publications > > http://www.perl.com/pub/2012/04/perlunicook-decode-standard- > > filehandles-as-utf-8.html which advice to use PerlIO on stderr. > > > > Also there is a pragma 'use open :std' in the CORE.

Mon Apr 08 10:11:29 2013 The RT System itself - Status changed from 'resolved' to 'open'

Sat Apr 27 08:51:27 2013 victor [...] vsespb.ru - Correspondence added

From:

victor [...] vsespb.ru

Hello. I've submited perlbug https://rt.perl.org/rt3//Public/Bug/Display.html?id=117537 one of possible suggested workaround for now is to use $PerlIO::encoding::fallback = Encode::FB_QUIET On Mon Apr 08 18:03:42 2013, DANKOGAI wrote: Show quoted text

> PerlIO handlers like ':utf8' are okay because they do not modify data. > Some are not since they do modify data (and more importantly, its > length) ':encoding()' is exactly that. > > I agree we should document that but it should be more documented in > PerlIO part... > > Dan > > On Mon Apr 08 09:50:56 2013, vsespb wrote:

> > Hm. > > > > Is it documented ? > > > > How then stacktrace (croak/carp/die) with non-ASCII data supposed to > > work ? > > > > There are some publications > > http://www.perl.com/pub/2012/04/perlunicook-decode-standard- > > filehandles-as-utf-8.html which advice to use PerlIO on stderr. > > > > Also there is a pragma 'use open :std' in the CORE.

Mon Aug 20 07:23:17 2018 SREZIC [...] cpan.org - Dependency on perl ticket #117537 added

Mon Aug 20 07:25:09 2018 SREZIC [...] cpan.org - Cc SREZIC added

Mon Aug 27 11:07:51 2018 SREZIC [...] cpan.org - Correspondence added

On 2013-04-07 16:42:59, vsespb wrote: Show quoted text

> perl -MEncode -e 'print Encode->VERSION' > 2.42_01 > > > The following commands hang with 50% CPU on > FreeBSD 9.1 (This is perl 5, version 14, subversion 2 (v5.14.2) built > for amd64-freebsd > ) > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", > "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}\x{43a}")' > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}\x{440}")' > > > and the following work fine: > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", "\x{410}\x{432}\x{442}\x{43e}\x{43f}\x{430}")' > > perl -MEncode -MCarp -e 'binmode STDERR, ":encoding(koi8-r)"; confess > encode("utf-8", "\x{440}\x{43a}")' > > > $ locale > LANG= > LC_CTYPE="C" > LC_COLLATE="C" > LC_TIME="C" > LC_NUMERIC="C" > LC_MONETARY="C" > LC_MESSAGES="C" > LC_ALL= > > also, cannot reproduce under linux > > possibly related to #84282

I have a similar problem (see attached script), but I can reproduce it on a Linux system (Mint 18). Don't make the filename of the test script shorter; this makes the warning message shorter and may not exhibit the problem anymore (or put some more X's into the test string). It does not seem to be related to a particular perl version (I see the problem with 5.12.3 and 5.28.0), but a bisect in Encode shows that the problem started with this commit 31d16654f4cb56f88dc7a01ee1897f4ec8318c1a is the first bad commit commit 31d16654f4cb56f88dc7a01ee1897f4ec8318c1a Author: dankogai <dankogai@d0d07461-0603-4401-acd4-de1884942a52> Date: Fri Dec 31 22:59:40 2010 +0000 VERSION 2.42 Trying the $PerlIO::encoding::fallback = Encode::FB_QUIET; workaround did not work for me. What did work is to replace utf-8 by utf8. Probably a fix is hard, but it would be nice if there was a paragraph about this problem (and the possible workaround using "utf8") in the BUGS or CAVEATS section of encoding.pod.

Subject:

broken-unicode-print-hangs-minimal.t

#!/usr/bin/perl use strict; use warnings; use utf8; use POSIX; use Test::More 'no_plan'; my $x = <<'EOF'; XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXÐ¨á¾î áêà¯Òá¾î£áêà¯Ó¨á¾î¦ áêà¯Õá¾îªáêà¯Ö¨á¾î áê¡à¯Øá¾î±áê£à¯Ù¨á¾î´ áê¤à¯Ûá¾î¸áê¥à¯Ü¨á¾î» áê¦à¯Þá¾pá§ê§à¯ß¨á¾Åá§ê¨à¯à¡á¾È°á§ê¨¤à¯à¢¨á¾Ìá§ê©¤à¯à¤á¾Ï°á§êªà¯à¥¨á¾Óá§ê«à¯à§á¾Ö°á§ê¬à¯à¨¨á¾Úá§êà¯àªá¾Ý°á§ê®à¯à«¨á¾à¡á§ê¯à¯àá¾à¤°á§ê°à¯à®¨á¾à¨á§ê²à¯à°á¾î» à½ï¤È¢ãº¨à¯¤Ý°È¬ï¤È¢ã¶à¯¤á½á¤ï¤È¢ã®¨à¯¤æá«ï¤È¢ã¨¨à¯¤æáï¤È¢ãà¯¤æáïÈ¢ã¨¨Èæ±á«ïÈ¢ãà¯¤æ á«ïÈ¢ãà¯¤æ á«ïÈ¢ç³à¬¢ê¨áïÈ¢à±¨à¯ê¤°áä¿Èèà¥á à¥ä¾Èèà¥î» á¹ä½Èã¡Èê¸à²ä¼Èã¨Èê à²ä»Èã¨Èì áäºÈãÈã áä¹Èã¨Èì¨áä¸Èá¶Èìáä·ÈáªÈì¸áä¶Èá¨Èì´ áäµÈáÈì±áä´Èç¶Èé°áä³Èè´¨Ç½ì£á³ä²Èï®¨à¯¤éáä±Èï·¨ EOF use Encode; $PerlIO::encoding::fallback = Encode::FB_QUIET; my $sigset = POSIX::SigSet->new(SIGALRM); # normal signal handler cannot be used because of safe signals my $sigaction = POSIX::SigAction->new(sub { die "Timeout" }, $sigset); alarm(3); binmode STDERR, ':encoding(utf-8)'; # does not hang if :encoding(utf8) is used print STDERR "<$x>\n"; alarm(0); pass "Does not hang"; __END__