Bug #19913 for CGI: param() returns bytes rather than a perl utf-8 string

Thanks, i have started using the decode_utf8() function. Something strange in the module CGI/Util.pm, is that when a character of a parameter is encoded as a numeric unicode sequence, then the returned perl string will be marked as utf-8 (in latest version of the CGI module). Yves. "Burak Gürsoy via RT" <bug-CGI.pm@rt.cpan.org> 15/06/2006 20:13 Please respond to bug-CGI.pm@rt.cpan.org To yves.lejeune@kodak.com cc Subject [rt.cpan.org #19913] param() returns bytes rather than a perl utf-8 string <URL: http://rt.cpan.org/Ticket/Display.html?id=19913 > You have to do that manually: use Encode; my $foo = decode utf8 => $cgi->param('foo'); On Thu Jun 15 05:08:17 2006, guest wrote: Show quoted text
using Show quoted text

I'm always scared about touching the unicode stuff because I don't fully understand it. However, give this version of CGI a try -- when it detects that charset is set to "utf-8" (all lowercase with the dash), it will automatically call decode() on the CGI values when it first parses them. Lincoln On Thu Jul 20 05:23:42 2006, FUZZ wrote: Show quoted text

Sorry, that version didn't work at all. Please try this version out. It runs decode() everytime you call param() so there is same overhead. Lincoln

You are probably right. Note that in my original bug report, my conclusion was: "I am not sure this would be considered as a bug by everybody, but it would be nice to have at least a note in the perldoc" I think that now I understand what was inconsistent in my analysis: CGI can not guess the relationship between the form using UTF-8 parameters, and the parameters of the resulting POST or GET request. CGI can only process the request parameters in the general case, i.e. as a string of bytes. The knowledge that request parameters are UTF-8 strings can only rely in the application code. At best there could be an optional feature, something like an option "all_request_parameters_are_utf8_strings", that would do the job in this specific context. Setting this option would prevent for instance to handle the upload of binary files. Best regards, Yves Lejeune. "Marian Ďurkovič via RT" <bug-CGI.pm@rt.cpan.org> wrote on 22/08/2007 09:45:36: Show quoted text
problems. Show quoted text

Attached patch against 3.45 includes test case and fix for the "double UTF-8 decode()" scenario described in this ticket. Basically it skips the Encode::decode() call if Encode::is_utf8() for the value is true (its already flagged UTF-8). Let me know if you have any questions.

diff --git a/lib/CGI.pm b/lib/CGI.pm index cacb03a..1b90a6a 100644 --- a/lib/CGI.pm +++ b/lib/CGI.pm @@ -455,12 +455,23 @@ sub param { if ($PARAM_UTF8) { eval "require Encode; 1;" unless Encode->can('decode'); # bring in these functions - @result = map {ref $_ ? $_ : Encode::decode(utf8=>$_) } @result; + @result = map {ref $_ ? $_ : $self->_decode_utf8($_) } @result; } return wantarray ? @result : $result[0]; } +sub _decode_utf8 { + my ($self, $val) = @_; + + if (Encode::is_utf8($val)) { + return $val; + } + else { + return Encode::decode(utf8 => $val); + } +} + sub self_or_default { return @_ if defined($_[0]) && (!ref($_[0])) &&($_[0] eq 'CGI'); unless (defined($_[0]) && diff --git a/t/utf8.t b/t/utf8.t new file mode 100644 index 0000000..8b5ad23 --- /dev/null +++ b/t/utf8.t @@ -0,0 +1,34 @@ +##!./perl -wT + +use strict; +use utf8; +use lib qw(t/lib); + +# Due to a bug in older versions of MakeMaker & Test::Harness, we must +# ensure the blib's are in @INC, else we might use the core CGI.pm +use lib qw(blib/lib blib/arch); + +use Test::More tests => 7; +use Encode; + +use_ok( 'CGI' ); + +ok( my $q = CGI->new(), 'create a new CGI object' ); + +$CGI::PARAM_UTF8 = 1; + +my $data = 'áéíóúµ'; +ok Encode::is_utf8($data), "created UTF-8 encoded data string"; + +# now set the param. +$q->param(data => $data); + +# if param() runs the data through Encode::decode(), this will fail. +is $q->param('data'), $data; + +# make sure setting bytes decodes properly +my $bytes = Encode::encode(utf8 => $data); +ok !Encode::is_utf8($bytes), "converted UTF-8 to bytes"; +$q->param(data => $bytes); +is $q->param('data'), $data; +ok Encode::is_utf8($q->param('data')), 'param() decoded UTF-8';