Bug #51204 for Encode: Callback CHECK not supported for UTF-8 decoder/encoder

Sun Nov 08 06:14:43 2009 GAAS [...] cpan.org - Ticket created

Subject:

Callback CHECK not supported for UTF-8 decoder/encoder

I was surprised how much time I ended up spending debugging why I could not get: Encode::decode("UTF-8", $octets, sub { sprintf "%%%02X", shift }) to work. My program behaved very oddly and the behaviour changed in random ways. Turns out Encode just sliently turn the CODE address into an integer and treat that as flags to decide how to behave. Very confusing. The least that should happen is that the decoder croaks when a CODE reference is passed in, but I would be much happier if such callbacks could simply be made to work.

Sun Nov 08 06:18:02 2009 GAAS [...] cpan.org - Correspondence added

Patch to make Encode::decode("UTF-8", $bytes, sub {}) croak.

From 32ca48a2d4d46aa42855266f4e37da8a7342a92d Mon Sep 17 00:00:00 2001 From: Gisle Aas <gisle@aas.no> Date: Sun, 8 Nov 2009 12:15:58 +0100 Subject: [PATCH] Make the UTF8 encoder croak when callback CHECK is passed in --- Encode.xs | 22 ++++++++++++++++++---- 1 files changed, 18 insertions(+), 4 deletions(-) diff --git a/Encode.xs b/Encode.xs index e5f4c9a..e9ccd3f 100644 --- a/Encode.xs +++ b/Encode.xs @@ -401,19 +401,26 @@ MODULE = Encode PACKAGE = Encode::utf8 PREFIX = Method_ PROTOTYPES: DISABLE void -Method_decode_xs(obj,src,check = 0) +Method_decode_xs(obj,src,check_sv = &PL_sv_no) SV * obj SV * src -int check +SV * check_sv PREINIT: STRLEN slen; U8 *s; U8 *e; SV *dst; bool renewed = 0; + int check; CODE: { dSP; ENTER; SAVETMPS; + if (SvROK(check_sv)) { + croak("UTF-8 decoder doesn't support callback CHECK"); + } + else { + check = SvIV(check_sv); + } if (src == &PL_sv_undef) src = newSV(0); s = (U8 *) SvPV(src, slen); e = (U8 *) SvEND(src); @@ -464,18 +471,25 @@ CODE: } void -Method_encode_xs(obj,src,check = 0) +Method_encode_xs(obj,src,check_sv = &PL_sv_no) SV * obj SV * src -int check +SV * check_sv PREINIT: STRLEN slen; U8 *s; U8 *e; SV *dst; bool renewed = 0; + int check; CODE: { + if (SvROK(check_sv)) { + croak("UTF-8 encoder doesn't support callback CHECK"); + } + else { + check = SvIV(check_sv); + } if (src == &PL_sv_undef) src = newSV(0); s = (U8 *) SvPV(src, slen); e = (U8 *) SvEND(src); -- 1.6.2.95.g934f7

Sun Nov 08 06:18:03 2009 GAAS [...] cpan.org - Status changed from 'new' to 'open'

Mon Nov 16 03:49:03 2009 DANKOGAI [...] cpan.org - Correspondence added

Thanks, applied in my repo. VERSION++ soon. Dan the Maintainer Thereof On Sun Nov 08 06:18:02 2009, GAAS wrote: Show quoted text

> Patch to make Encode::decode("UTF-8", $bytes, sub {}) croak.

Mon Nov 16 03:49:05 2009 DANKOGAI [...] cpan.org - Status changed from 'open' to 'resolved'

Mon Nov 23 18:17:14 2009 u-suke [...] kawa.net - Correspondence added

Hi, the ticket seems closed though. Some of my modules expected that this wouldn't croak but be simply ignored. I would be much happier if such callbacks could work, too. :-) I tested under Encode.pm 2.23: perl -MEncode -e 'print Encode::decode("UTF-8", "\x80", Encode::FB_XMLCREF), "\n"'  => CHECK values works as single ¥x80 octets is malformed in UTF-8. I guess this is one of cases that CHECK is needed for decode(). perl -MEncode -e 'print Encode::decode("UTF-8", "\x80", sub { sprintf "%%%02X", shift }), "\n"'  => CHECK coderef is ignored and doesn't work. I think this should work however. This invokes "UTF-8 decoder doesn't support callback CHECK" under Encode 2.38. Thank you both for maintaining the great module anyway. Ref: http://kawa.at.webry.info/200911/article_12.html

Mon Nov 23 18:17:14 2009 The RT System itself - Status changed from 'resolved' to 'open'

Thu Nov 26 04:28:22 2009 DANKOGAI [...] cpan.org - Correspondence added

Fixed in Version 2.39. Thanks for your insight. Indeed, it's useful on decode. For encode, it has no use since it always succeeds. Dan the Encode Maintainer On Mon Nov 23 18:17:14 2009, KAWASAKI wrote: Show quoted text

> Hi, the ticket seems closed though. > > Some of my modules expected that this wouldn't croak but be simply > ignored. > I would be much happier if such callbacks could work, too. :-) > > I tested under Encode.pm 2.23: > > perl -MEncode -e 'print Encode::decode("UTF-8", "\x80", > Encode::FB_XMLCREF), "\n"' >  > > => CHECK values works as single ¥x80 octets is malformed in UTF-8. > I guess this is one of cases that CHECK is needed for decode(). > > perl -MEncode -e 'print Encode::decode("UTF-8", "\x80", sub { sprintf > "%%%02X", shift }), > "\n"' >  > > => CHECK coderef is ignored and doesn't work. I think this should work > however. > This invokes "UTF-8 decoder doesn't support callback CHECK" under > Encode 2.38. > > Thank you both for maintaining the great module anyway. > > Ref: > http://kawa.at.webry.info/200911/article_12.html

Thu Nov 26 04:28:23 2009 DANKOGAI [...] cpan.org - Status changed from 'open' to 'resolved'