Skip Menu |

This queue is for tickets about the Encode CPAN distribution.

Report information
The Basics
Id: 100427
Status: resolved
Priority: 0/
Queue: Encode

People
Owner: Nobody in particular
Requestors: ppisar [...] redhat.com
Cc: pali [...] cpan.org
AdminCc:

Bug Information
Severity: (no value)
Broken in: 2.64
Fixed in: (no value)



Subject: use encoding "foo", Filter=>1; does not raise an exception if Filter::Util::Call cannot be used
Running Encode tests without Filter::Util::Call module available in the system makes t/jperl.t to fail: $ perl -Iblib/{lib,arch} t/jperl.t 1..15 Unrecognized character \x{fffd}; marked by <-- HERE after our $<-- HERE near column 10 at t/jperl.t line 90. # Looks like your test exited with 2 before it could output anything. The test does: { use encoding "euc-jp", Filter=>1; ok(1, "Filter on"); use utf8; no strict 'vars'; # fools # doesn't work w/ "my" as of this writing. # because of buggy strict.pm and utf8.pm our $人 = 2; # ^^U+4eba, "human" in CJK ideograph $人++; # a child is born *people = \$人; is ($people, 3, "Filter:utf8 identifier"); no encoding; ok(1, "Filter off"); } Obviously the first ok() is there to confirm the "use encoding ..., Filter=>1" pragma succeeded. However the encoding.pm does: unless ( $arg{Filter} ) { ... } else { defined( ${^ENCODING} ) and undef ${^ENCODING}; # implicitly 'use utf8' require utf8; # to fetch $utf8::hint_bits; $^H |= $utf8::hint_bits; eval { require Filter::Util::Call; Filter::Util::Call->import; filter_add( sub { my $status = filter_read(); if ( $status > 0 ) { $_ = $enc->decode( $_, 1 ); DEBUG and warn $_; } $status; } ); }; $@ eq '' and DEBUG and warn "Filter installed"; } The code hides any exception by the eval, so the above-mentioned first test should succeed all the time. I know the "use encoding" is deprecated, so I do not push for working on this issue or changing behavior. I'm just curious why the encode.pm does pass the exception. (This can be handy if the character set identifier is uknown or the source byte stream is not well-formed for the specified character set.)
The logic therein was written circa 2001. Even I don't remember. At any rate now that Unicode is de-facto standard for new scripts to be written, encoding.pm is no longer needed… Dan On Tue Nov 18 10:18:10 2014, ppisar wrote: Show quoted text
> Running Encode tests without Filter::Util::Call module available in > the system makes t/jperl.t to fail: > > $ perl -Iblib/{lib,arch} t/jperl.t > 1..15 > Unrecognized character \x{fffd}; marked by <-- HERE after our $<-- > HERE near column 10 at t/jperl.t line 90. > # Looks like your test exited with 2 before it could output anything. > > The test does: > > { > use encoding "euc-jp", Filter=>1; > ok(1, "Filter on"); > use utf8; > no strict 'vars'; # fools > # doesn't work w/ "my" as of this writing. > # because of buggy strict.pm and utf8.pm > our $人 = 2; > # ^^U+4eba, "human" in CJK ideograph > $人++; # a child is born > *people = \$人; > is ($people, 3, "Filter:utf8 identifier"); > no encoding; > ok(1, "Filter off"); > } > > Obviously the first ok() is there to confirm the "use encoding ..., > Filter=>1" pragma succeeded. However the encoding.pm does: > > unless ( $arg{Filter} ) { > ... > } > else { > defined( ${^ENCODING} ) and undef ${^ENCODING}; > > # implicitly 'use utf8' > require utf8; # to fetch $utf8::hint_bits; > $^H |= $utf8::hint_bits; > eval { > require Filter::Util::Call; > Filter::Util::Call->import; > filter_add( > sub { > my $status = filter_read(); > if ( $status > 0 ) { > $_ = $enc->decode( $_, 1 ); > DEBUG and warn $_; > } > $status; > } > ); > }; > $@ eq '' and DEBUG and warn "Filter installed"; > } > > The code hides any exception by the eval, so the above-mentioned first > test should succeed all the time. > > I know the "use encoding" is deprecated, so I do not push for working > on this issue or changing behavior. I'm just curious why the encode.pm > does pass the exception. (This can be handy if the character set > identifier is uknown or the source byte stream is not well-formed for > the specified character set.)
That code should not be executed in eval, but rather directly with throwing exceptions back to that caller. "use encoding" must ensure that used filter is installed and used. Otherwise it is fatal error as following code is written in different encoding... Dan, will you fix it and remove eval wrapper? 'use encoding' is removed in Perl 5.26, but not 'use encoding, Filter' which is still working. See: https://metacpan.org/pod/release/XSAWYERX/perl-5.26.0/pod/perldelta.pod#The-${^ENCODING}-facility-has-been-removed Therefore Filter encoding should be fixed. On Thu Nov 27 03:14:00 2014, DANKOGAI wrote: Show quoted text
> The logic therein was written circa 2001. Even I don't remember. At > any rate now that Unicode is de-facto standard for new scripts to be > written, encoding.pm is no longer needed… > > Dan > > On Tue Nov 18 10:18:10 2014, ppisar wrote:
> > Running Encode tests without Filter::Util::Call module available in > > the system makes t/jperl.t to fail: > > > > $ perl -Iblib/{lib,arch} t/jperl.t > > 1..15 > > Unrecognized character \x{fffd}; marked by <-- HERE after our > > $<-- > > HERE near column 10 at t/jperl.t line 90. > > # Looks like your test exited with 2 before it could output anything. > > > > The test does: > > > > { > > use encoding "euc-jp", Filter=>1; > > ok(1, "Filter on"); > > use utf8; > > no strict 'vars'; # fools > > # doesn't work w/ "my" as of this writing. > > # because of buggy strict.pm and utf8.pm > > our $人 = 2; > > # ^^U+4eba, "human" in CJK ideograph > > $人++; # a child is born > > *people = \$人; > > is ($people, 3, "Filter:utf8 identifier"); > > no encoding; > > ok(1, "Filter off"); > > } > > > > Obviously the first ok() is there to confirm the "use encoding ..., > > Filter=>1" pragma succeeded. However the encoding.pm does: > > > > unless ( $arg{Filter} ) { > > ... > > } > > else { > > defined( ${^ENCODING} ) and undef ${^ENCODING}; > > > > # implicitly 'use utf8' > > require utf8; # to fetch $utf8::hint_bits; > > $^H |= $utf8::hint_bits; > > eval { > > require Filter::Util::Call; > > Filter::Util::Call->import; > > filter_add( > > sub { > > my $status = filter_read(); > > if ( $status > 0 ) { > > $_ = $enc->decode( $_, 1 ); > > DEBUG and warn $_; > > } > > $status; > > } > > ); > > }; > > $@ eq '' and DEBUG and warn "Filter installed"; > > } > > > > The code hides any exception by the eval, so the above-mentioned > > first > > test should succeed all the time. > > > > I know the "use encoding" is deprecated, so I do not push for working > > on this issue or changing behavior. I'm just curious why the > > encode.pm > > does pass the exception. (This can be handy if the character set > > identifier is uknown or the source byte stream is not well-formed for > > the specified character set.)
In attachment is a patch with proposed fix. ppisar, can you test this patch if it fixes problem which you reported? On Fri Jun 02 04:45:06 2017, PALI wrote: Show quoted text
> That code should not be executed in eval, but rather directly with > throwing exceptions back to that caller. "use encoding" must ensure > that used filter is installed and used. Otherwise it is fatal error as > following code is written in different encoding... > > Dan, will you fix it and remove eval wrapper? 'use encoding' is > removed in Perl 5.26, but not 'use encoding, Filter' which is still > working. See: https://metacpan.org/pod/release/XSAWYERX/perl- > 5.26.0/pod/perldelta.pod#The-${^ENCODING}-facility-has-been-removed > > Therefore Filter encoding should be fixed. > > On Thu Nov 27 03:14:00 2014, DANKOGAI wrote:
> > The logic therein was written circa 2001. Even I don't remember. At > > any rate now that Unicode is de-facto standard for new scripts to be > > written, encoding.pm is no longer needed… > > > > Dan > > > > On Tue Nov 18 10:18:10 2014, ppisar wrote:
> > > Running Encode tests without Filter::Util::Call module available in > > > the system makes t/jperl.t to fail: > > > > > > $ perl -Iblib/{lib,arch} t/jperl.t > > > 1..15 > > > Unrecognized character \x{fffd}; marked by <-- HERE after our > > > $<-- > > > HERE near column 10 at t/jperl.t line 90. > > > # Looks like your test exited with 2 before it could output > > > anything. > > > > > > The test does: > > > > > > { > > > use encoding "euc-jp", Filter=>1; > > > ok(1, "Filter on"); > > > use utf8; > > > no strict 'vars'; # fools > > > # doesn't work w/ "my" as of this writing. > > > # because of buggy strict.pm and utf8.pm > > > our $人 = 2; > > > # ^^U+4eba, "human" in CJK ideograph > > > $人++; # a child is born > > > *people = \$人; > > > is ($people, 3, "Filter:utf8 identifier"); > > > no encoding; > > > ok(1, "Filter off"); > > > } > > > > > > Obviously the first ok() is there to confirm the "use encoding ..., > > > Filter=>1" pragma succeeded. However the encoding.pm does: > > > > > > unless ( $arg{Filter} ) { > > > ... > > > } > > > else { > > > defined( ${^ENCODING} ) and undef ${^ENCODING}; > > > > > > # implicitly 'use utf8' > > > require utf8; # to fetch $utf8::hint_bits; > > > $^H |= $utf8::hint_bits; > > > eval { > > > require Filter::Util::Call; > > > Filter::Util::Call->import; > > > filter_add( > > > sub { > > > my $status = filter_read(); > > > if ( $status > 0 ) { > > > $_ = $enc->decode( $_, 1 ); > > > DEBUG and warn $_; > > > } > > > $status; > > > } > > > ); > > > }; > > > $@ eq '' and DEBUG and warn "Filter installed"; > > > } > > > > > > The code hides any exception by the eval, so the above-mentioned > > > first > > > test should succeed all the time. > > > > > > I know the "use encoding" is deprecated, so I do not push for > > > working > > > on this issue or changing behavior. I'm just curious why the > > > encode.pm > > > does pass the exception. (This can be handy if the character set > > > identifier is uknown or the source byte stream is not well-formed > > > for > > > the specified character set.)
Subject: 0001-Propagate-fatal-errors-from-the-encoding-pragma-back.patch
From 020c0a8a6e9a3858d78c672e59e1f6babfd5627f Mon Sep 17 00:00:00 2001 From: Pali <pali@cpan.org> Date: Fri, 29 Sep 2017 14:31:33 +0200 Subject: [PATCH] Propagate fatal errors from the encoding pragma back to the caller In some cases (e.g. when Filter::Util::Call module is not available), encoding pragma would not work. So caller should get fatal error message instead of continuous execution of the wrong code. --- encoding.pm | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/encoding.pm b/encoding.pm index 7cd9eb2..651a158 100644 --- a/encoding.pm +++ b/encoding.pm @@ -176,7 +176,7 @@ sub import { # implicitly 'use utf8' require utf8; # to fetch $utf8::hint_bits; $^H |= $utf8::hint_bits; - eval { + require Filter::Util::Call; Filter::Util::Call->import; filter_add( @@ -189,8 +189,6 @@ sub import { $status; } ); - 1; - } and DEBUG and warn "Filter installed"; } defined ${^UNICODE} and ${^UNICODE} != 0 and return 1; for my $h (qw(STDIN STDOUT)) { @@ -200,20 +198,14 @@ sub import { Carp::croak( "encoding: Unknown encoding for $h, '$arg{$h}'"); } - eval { binmode( $h, ":raw :encoding($arg{$h})" ) }; + binmode( $h, ":raw :encoding($arg{$h})" ); } else { unless ( exists $arg{$h} ) { - eval { no warnings 'uninitialized'; binmode( $h, ":raw :encoding($name)" ); - }; } } - if ($@) { - require Carp; - Carp::croak($@); - } } return 1; # I doubt if we need it, though } -- 2.11.0
Subject: Re: [rt.cpan.org #100427] use encoding "foo", Filter=>1; does not raise an exception if Filter::Util::Call cannot be used
Date: Fri, 29 Sep 2017 15:15:58 +0200
To: Pali via RT <bug-Encode [...] rt.cpan.org>
From: Petr Pisar <ppisar [...] redhat.com>
On Fri, Sep 29, 2017 at 08:35:37AM -0400, Pali via RT wrote: Show quoted text
> In attachment is a patch with proposed fix. ppisar, can you test this patch > if it fixes problem which you reported? >
Latest Encode-2.92 skips the test on Perl 5.26. When I run the test on different host with Perl 5.22.3, I can reproduce the bug and after applying your patch, the test fails with expected error now: $ perl -Iblib/{lib,arch} t/jperl.t 1..15 Can't locate Filter/Util/Call.pm in @INC (you may need to install the Filter::Util::Call module) (@INC contains: blib/lib blib/arch /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at blib/lib/encoding.pm line 180. BEGIN failed--compilation aborted at t/jperl.t line 88. # Looks like your test exited with 2 before it could output anything. So I think it fixes the problem. Next step could be adding a dependency on Filter::Util::Call into META. Thank you for looking into this issue. -- Petr
On Fri Sep 29 09:16:00 2017, ppisar wrote: Show quoted text
> On Fri, Sep 29, 2017 at 08:35:37AM -0400, Pali via RT wrote:
> > In attachment is a patch with proposed fix. ppisar, can you test this > > patch > > if it fixes problem which you reported? > >
> Latest Encode-2.92 skips the test on Perl 5.26. When I run the test on > different host with Perl 5.22.3, I can reproduce the bug and after > applying > your patch, the test fails with expected error now: > > $ perl -Iblib/{lib,arch} t/jperl.t 1..15 > Can't locate Filter/Util/Call.pm in @INC (you may need to install the > Filter::Util::Call module) (@INC contains: blib/lib blib/arch > /usr/local/lib64/perl5 /usr/local/share/perl5 > /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl > /usr/lib64/perl5 /usr/share/perl5 .) at blib/lib/encoding.pm line 180. > BEGIN failed--compilation aborted at t/jperl.t line 88. > # Looks like your test exited with 2 before it could output anything. > > So I think it fixes the problem.
Ok. Show quoted text
> Next step could be adding a dependency on Filter::Util::Call into > META.
Looking at corelist... $ corelist Filter::Util::Call Data for 2017-01-14 Filter::Util::Call was first released with perl v5.7.3 ... it tells me that Filter::Util::Call is part of perl itself since v5.7.3. So this is looks like a problem with your installation of perl, you should have Filter::Util::Call available. But adding perl 5.8 into META seems like a good idea. It should fix also problem with Filter::Util::Call.
On Pia Sep 29 09:21:26 2017, PALI wrote: Show quoted text
> On Fri Sep 29 09:16:00 2017, ppisar wrote:
> > On Fri, Sep 29, 2017 at 08:35:37AM -0400, Pali via RT wrote:
> > > In attachment is a patch with proposed fix. ppisar, can you test > > > this > > > patch > > > if it fixes problem which you reported? > > >
> > Latest Encode-2.92 skips the test on Perl 5.26. When I run the test > > on > > different host with Perl 5.22.3, I can reproduce the bug and after > > applying > > your patch, the test fails with expected error now: > > > > $ perl -Iblib/{lib,arch} t/jperl.t 1..15 > > Can't locate Filter/Util/Call.pm in @INC (you may need to install the > > Filter::Util::Call module) (@INC contains: blib/lib blib/arch > > /usr/local/lib64/perl5 /usr/local/share/perl5 > > /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl > > /usr/lib64/perl5 /usr/share/perl5 .) at blib/lib/encoding.pm line > > 180. > > BEGIN failed--compilation aborted at t/jperl.t line 88. > > # Looks like your test exited with 2 before it could output anything. > > > > So I think it fixes the problem.
> > Ok.
I created pull request with this patch: https://github.com/dankogai/p5-encode/pull/123
Pulled. Dan the Maintainer Thereof On Sat Sep 30 08:28:31 2017, PALI wrote: Show quoted text
> On Pia Sep 29 09:21:26 2017, PALI wrote:
> > On Fri Sep 29 09:16:00 2017, ppisar wrote:
> > > On Fri, Sep 29, 2017 at 08:35:37AM -0400, Pali via RT wrote:
> > > > In attachment is a patch with proposed fix. ppisar, can you test > > > > this > > > > patch > > > > if it fixes problem which you reported? > > > >
> > > Latest Encode-2.92 skips the test on Perl 5.26. When I run the test > > > on > > > different host with Perl 5.22.3, I can reproduce the bug and after > > > applying > > > your patch, the test fails with expected error now: > > > > > > $ perl -Iblib/{lib,arch} t/jperl.t 1..15 > > > Can't locate Filter/Util/Call.pm in @INC (you may need to install the > > > Filter::Util::Call module) (@INC contains: blib/lib blib/arch > > > /usr/local/lib64/perl5 /usr/local/share/perl5 > > > /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl > > > /usr/lib64/perl5 /usr/share/perl5 .) at blib/lib/encoding.pm line > > > 180. > > > BEGIN failed--compilation aborted at t/jperl.t line 88. > > > # Looks like your test exited with 2 before it could output anything. > > > > > > So I think it fixes the problem.
> > > > Ok.
> > I created pull request with this patch: > https://github.com/dankogai/p5-encode/pull/123 >