Skip Menu |

This queue is for tickets about the EJS-Template CPAN distribution.

Report information
The Basics
Id: 104753
Status: resolved
Priority: 0/
Queue: EJS-Template

People
Owner: MAHIRO [...] cpan.org
Requestors: jozef.mojzis [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: Inconsistent utf8 handling...
Date: Thu, 28 May 2015 17:27:08 +0200
To: bug-EJS-Template [...] rt.cpan.org
From: Jozef Mojzis <jozef.mojzis [...] gmail.com>
Strange unicode problems. Used engine: "JE". Try the following template "test.ejs" (utf8 file). Show quoted text
--- cut here --- <% var intempl = "čšňž"; var joined = intempl + fromperl; %> Length of the intempl: <%= intempl.length %> Length of the fromperl: <%= fromperl.length %> Length of the joined: <%= joined.length %> <% for (var i = 0; i < joined.length; i++) { %> at position <%= i %> =<%= joined.charAt(i) %>= <% } %>
--- cut here --- and run it with the following script (also utf8). use 5.014; #<-- use strict; use feature 'unicode_strings'; use warnings; use utf8; #this source contains utf8 encoded characters use Path::Tiny; use EJS::Template; binmode STDOUT,':utf8'; #slurp utf8 file my $input = path("test.ejs")->slurp_utf8; say "-- [ input ]", "-"x40; say $input; #parse and print the code my $code; EJS::Template->parse(\$input, \$code); say "-- [ code ]", "-"x40; say $code; #run the JS my $out; EJS::Template->execute(\$code, {fromperl => 'čšňž'}, \$out); say "-- [ output ]", "-"x40; say $out; In the rendered output is - correctly handled the variable declared in the javascript (the variable intempl - e.g. the JE-engine knows how to handle wide chars) - but the variable coming from perl isn't handled correctly (the passing method is probably bugged somewhat). Ps: both variables (intempl and fromperl) (should) contains: U+0010D č LATIN SMALL LETTER C WITH CARON U+00161 š LATIN SMALL LETTER S WITH CARON U+00148 ň LATIN SMALL LETTER N WITH CARON U+0017E ž LATIN SMALL LETTER Z WITH CARON
Thanks for reporting with the details. It does look strange. I'm looking into it. On Thu, 28 May 2015 15:27:29 GMT, jomo wrote: Show quoted text
> Strange unicode problems. Used engine: "JE". > > Try the following template "test.ejs" (utf8 file). > > --- cut here --- > <% > var intempl = "čšňž"; > var joined = intempl + fromperl; > %> > > Length of the intempl: <%= intempl.length %> > Length of the fromperl: <%= fromperl.length %> > Length of the joined: <%= joined.length %> > > <% for (var i = 0; i < joined.length; i++) { %> > at position <%= i %> =<%= joined.charAt(i) %>= > <% } %> > --- cut here --- > > > and run it with the following script (also utf8). > use 5.014; #<-- use strict; use feature 'unicode_strings'; > use warnings; > use utf8; #this source contains utf8 encoded characters > > use Path::Tiny; > use EJS::Template; > > binmode STDOUT,':utf8'; > > #slurp utf8 file > my $input = path("test.ejs")->slurp_utf8; > say "-- [ input ]", "-"x40; > say $input; > > > #parse and print the code > my $code; > EJS::Template->parse(\$input, \$code); > say "-- [ code ]", "-"x40; > say $code; > > #run the JS > my $out; > EJS::Template->execute(\$code, {fromperl => 'čšňž'}, \$out); > say "-- [ output ]", "-"x40; > say $out; > > > In the rendered output is > - correctly handled the variable declared in the javascript (the > variable intempl - e.g. the JE-engine knows how to handle wide chars) > - but the variable coming from perl isn't handled correctly (the > passing method is probably bugged somewhat). > > Ps: both variables (intempl and fromperl) (should) contains: > U+0010D č LATIN SMALL LETTER C WITH CARON > U+00161 š LATIN SMALL LETTER S WITH CARON > U+00148 ň LATIN SMALL LETTER N WITH CARON > U+0017E ž LATIN SMALL LETTER Z WITH CARON
Subject: [rt.cpan.org #104753] Inconsistent utf8 handling...
Date: Sat, 6 Jun 2015 15:49:27 +0200
To: bug-EJS-Template [...] rt.cpan.org
From: Jozef Mojzis <jozef.mojzis [...] gmail.com>
My hotfix is: change $ENCODE_UTF8 = 1 to $ENCODE_UTF8 = 0 but don't checked deeply the fix impact... Maybe you could check and fix the problem and release the fixed version? ;) e.g --- EJSorig/Template/JSAdapter/JE.pm 2015-06-06 15:42:51.000000000 +0200 +++ EJS/Template/JSAdapter/JE.pm 2015-06-06 15:40:32.000000000 +0200 @@ -14,7 +14,7 @@ use EJS::Template::Util qw(clean_text_ref); use Scalar::Util qw(reftype); -our $ENCODE_UTF8 = 1; +our $ENCODE_UTF8 = 0; our $SANITIZE_UTF8 = 0; our $FORCE_UNTAINT = 0; Show quoted text
> On 02 Jun 2015, at 16:55, Mahiro Ando via RT <bug-EJS-Template@rt.cpan.org> wrote: > > <URL: https://rt.cpan.org/Ticket/Display.html?id=104753 > > > Thanks for reporting with the details. It does look strange. I'm looking into it. > > On Thu, 28 May 2015 15:27:29 GMT, jomo wrote:
>> Strange unicode problems. Used engine: "JE". >> >> Try the following template "test.ejs" (utf8 file). >> >> --- cut here --- >> <% >> var intempl = "čšňž"; >> var joined = intempl + fromperl; >> %> >> >> Length of the intempl: <%= intempl.length %> >> Length of the fromperl: <%= fromperl.length %> >> Length of the joined: <%= joined.length %> >> >> <% for (var i = 0; i < joined.length; i++) { %> >> at position <%= i %> =<%= joined.charAt(i) %>= >> <% } %> >> --- cut here --- >> >> >> and run it with the following script (also utf8). >> use 5.014; #<-- use strict; use feature 'unicode_strings'; >> use warnings; >> use utf8; #this source contains utf8 encoded characters >> >> use Path::Tiny; >> use EJS::Template; >> >> binmode STDOUT,':utf8'; >> >> #slurp utf8 file >> my $input = path("test.ejs")->slurp_utf8; >> say "-- [ input ]", "-"x40; >> say $input; >> >> >> #parse and print the code >> my $code; >> EJS::Template->parse(\$input, \$code); >> say "-- [ code ]", "-"x40; >> say $code; >> >> #run the JS >> my $out; >> EJS::Template->execute(\$code, {fromperl => 'čšňž'}, \$out); >> say "-- [ output ]", "-"x40; >> say $out; >> >> >> In the rendered output is >> - correctly handled the variable declared in the javascript (the >> variable intempl - e.g. the JE-engine knows how to handle wide chars) >> - but the variable coming from perl isn't handled correctly (the >> passing method is probably bugged somewhat). >> >> Ps: both variables (intempl and fromperl) (should) contains: >> U+0010D č LATIN SMALL LETTER C WITH CARON >> U+00161 š LATIN SMALL LETTER S WITH CARON >> U+00148 ň LATIN SMALL LETTER N WITH CARON >> U+0017E ž LATIN SMALL LETTER Z WITH CARON
> > >
It's been a long time, but I've finally had chance to fix the utf8 issue to make JS adapters work as consistently as I can. Please find v0.08 with the fix. Thanks again for reporting and helping. On Sat, 6 Jun 2015 13:49:06 GMT, jomo wrote: Show quoted text
> My hotfix is: change > $ENCODE_UTF8 = 1 > to > $ENCODE_UTF8 = 0 > > but don't checked deeply the fix impact... > Maybe you could check and fix the problem and release the fixed > version? ;) > > e.g > --- EJSorig/Template/JSAdapter/JE.pm 2015-06-06 15:42:51.000000000 > +0200 > +++ EJS/Template/JSAdapter/JE.pm 2015-06-06 15:40:32.000000000 > +0200 > @@ -14,7 +14,7 @@ > use EJS::Template::Util qw(clean_text_ref); > use Scalar::Util qw(reftype); > > -our $ENCODE_UTF8 = 1; > +our $ENCODE_UTF8 = 0; > our $SANITIZE_UTF8 = 0; > our $FORCE_UNTAINT = 0; > >
> > On 02 Jun 2015, at 16:55, Mahiro Ando via RT <bug-EJS- > > Template@rt.cpan.org> wrote: > > > > <URL: https://rt.cpan.org/Ticket/Display.html?id=104753 > > > > > Thanks for reporting with the details. It does look strange. I'm > > looking into it. > > > > On Thu, 28 May 2015 15:27:29 GMT, jomo wrote:
> >> Strange unicode problems. Used engine: "JE". > >> > >> Try the following template "test.ejs" (utf8 file). > >> > >> --- cut here --- > >> <% > >> var intempl = "čšňž"; > >> var joined = intempl + fromperl; > >> %> > >> > >> Length of the intempl: <%= intempl.length %> > >> Length of the fromperl: <%= fromperl.length %> > >> Length of the joined: <%= joined.length %> > >> > >> <% for (var i = 0; i < joined.length; i++) { %> > >> at position <%= i %> =<%= joined.charAt(i) %>= > >> <% } %> > >> --- cut here --- > >> > >> > >> and run it with the following script (also utf8). > >> use 5.014; #<-- use strict; use feature 'unicode_strings'; > >> use warnings; > >> use utf8; #this source contains utf8 encoded characters > >> > >> use Path::Tiny; > >> use EJS::Template; > >> > >> binmode STDOUT,':utf8'; > >> > >> #slurp utf8 file > >> my $input = path("test.ejs")->slurp_utf8; > >> say "-- [ input ]", "-"x40; > >> say $input; > >> > >> > >> #parse and print the code > >> my $code; > >> EJS::Template->parse(\$input, \$code); > >> say "-- [ code ]", "-"x40; > >> say $code; > >> > >> #run the JS > >> my $out; > >> EJS::Template->execute(\$code, {fromperl => 'čšňž'}, \$out); > >> say "-- [ output ]", "-"x40; > >> say $out; > >> > >> > >> In the rendered output is > >> - correctly handled the variable declared in the javascript (the > >> variable intempl - e.g. the JE-engine knows how to handle wide > >> chars) > >> - but the variable coming from perl isn't handled correctly (the > >> passing method is probably bugged somewhat). > >> > >> Ps: both variables (intempl and fromperl) (should) contains: > >> U+0010D č LATIN SMALL LETTER C WITH CARON > >> U+00161 š LATIN SMALL LETTER S WITH CARON > >> U+00148 ň LATIN SMALL LETTER N WITH CARON > >> U+0017E ž LATIN SMALL LETTER Z WITH CARON
> > > > > >