Skip Menu |

Preferred bug tracker

Please visit the preferred bug tracker to report your issue.

This queue is for tickets about the Locale-Maketext CPAN distribution.

Report information
The Basics
Id: 46738
Status: resolved
Priority: 0/
Queue: Locale-Maketext

People
Owner: ferreira [...] shoo.cpan.org
Requestors: dmuey [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Critical
Broken in: (no value)
Fixed in: 1.15_01



Subject: Support readonly lexicon hashes (patch included)
If you tie() a lexicon to, say, a GDBM_File tied via & GDBM_READER; file the first call to maketext() is fatal. That is because once it compiles a value it re-stores it in the Lexicon hash. It is along the lines of this pseudo code: $gdbm_reader{$phrase} = $compile_value; # die()s The way to support this sort of hash is to provide a mechanism to not store the compiled strings back in the Lexicon The patched maktext() is in Locale::Maketext::Utils version 0.16 (w/ tests: t/04.use_external_lex_cache.t) but it'd be better suited for Locale::Maketext The way it works is this: You's init $lh->{'use_external_lex_cache'} = 1; Anytime something it is compiled it is stored in $lh- Show quoted text
>{_external_lex_cache} instead of the lexicon hash.
Subject: external_lex_cache.patch
--- Maketext.pm.orig 2009-06-05 11:42:02.000000000 -0500 +++ Maketext.pm 2009-06-08 10:03:26.000000000 -0500 @@ -129,10 +129,9 @@ my $phrase = shift; $handle->{'failure_lex'} ||= {}; - my $lex = $handle->{'failure_lex'}; - my $value; - $lex->{$phrase} ||= ($value = $handle->_compile($phrase)); + my $value = $handle->_compile($phrase); + $handle->{'failure_lex'}{$phrase} ||= $value; # ? respect 'use_external_lex_cache' || document not to set 'failure_lex' as a readonly type hash || ... ? # Dumbly copied from sub maketext: return ${$value} if ref($value) eq 'SCALAR'; @@ -179,34 +178,47 @@ my($handle, $phrase) = splice(@_,0,2); Carp::confess('No handle/phrase') unless (defined($handle) && defined($phrase)); - # Don't interefere with $@ in case that's being interpolated into the msg. local $@; # Look up the value: - my $value; - foreach my $h_r ( - @{ $isa_scan{ref($handle) || $handle} || $handle->_lex_refs } - ) { - DEBUG and warn "* Looking up \"$phrase\" in $h_r\n"; - if(exists $h_r->{$phrase}) { - DEBUG and warn " Found \"$phrase\" in $h_r\n"; - unless(ref($value = $h_r->{$phrase})) { - # Nonref means it's not yet compiled. Compile and replace. - $value = $h_r->{$phrase} = $handle->_compile($value); + if (exists $handle->{'_external_lex_cache'}{$phrase}) { + DEBUG and warn "* Using external lex cache version of \"$phrase\"\n"; + $value = $handle->{'_external_lex_cache'}{$phrase}; + } + else { + foreach my $h_r ( + @{ $isa_scan{ref($handle) || $handle} || $handle->_lex_refs } + ) { + DEBUG and warn "* Looking up \"$phrase\" in $h_r\n"; + if(exists $h_r->{$phrase}) { + DEBUG and warn " Found \"$phrase\" in $h_r\n"; + unless(ref($value = $h_r->{$phrase})) { + # Nonref means it's not yet compiled. Compile and replace. + if ($handle->{'use_external_lex_cache'}) { + $value = $handle->{'_external_lex_cache'}{$phrase} = $handle->_compile($value); + } + else { + $value = $h_r->{$phrase} = $handle->_compile($value); + } + } + last; } - last; - } - elsif($phrase !~ m/^_/s and $h_r->{'_AUTO'}) { - # it's an auto lex, and this is an autoable key! - DEBUG and warn " Automaking \"$phrase\" into $h_r\n"; - - $value = $h_r->{$phrase} = $handle->_compile($phrase); - last; + elsif($phrase !~ m/^_/s and $h_r->{'_AUTO'}) { + # it's an auto lex, and this is an autoable key! + DEBUG and warn " Automaking \"$phrase\" into $h_r\n"; + if ($handle->{'use_external_lex_cache'}) { + $value = $handle->{'_external_lex_cache'}{$phrase} = $handle->_compile($phrase); + } + else { + $value = $h_r->{$phrase} = $handle->_compile($phrase); + } + last; + } + DEBUG>1 and print " Not found in $h_r, nor automakable\n"; + # else keep looking } - DEBUG>1 and print " Not found in $h_r, nor automakable\n"; - # else keep looking } unless(defined($value)) {
this behavior could be implemented with an object lexicon. See https://rt.cpan.org/Ticket/Display.html?id=47315
On Thu Jun 25 09:45:58 2009, DMUEY wrote: Show quoted text
> this behavior could be implemented with an object lexicon. > > See https://rt.cpan.org/Ticket/Display.html?id=47315
On second thought, while that is true, this way is much less overhead if all you are wanting an object for is to gain this behavior
Locale/Maketext.pm would also need this: @@ -234,7 +234,7 @@ } last; } - elsif($phrase !~ m/^_/s and $h_r->{'_AUTO'}) { + elsif($phrase !~ m/^_/s and ($handle- Show quoted text
>{'use_external_lex_cache'} ? $handle->{'_external_lex_cache'}{'_AUTO'}
: $h_r->{'_AUTO'})) { # it's an auto lex, and this is an autoable key! DEBUG and warn " Automaking \"$phrase\" into $h_r\n"; Note to self: Locale/Maketext/Utils.pm @@ -62,9 +62,15 @@ } } - no strict 'refs'; - local ${ $lh->get_base_class() . '::Lexicon' }{'_AUTO'} = 1; - return $lh->maketext( $key, @args ); + if ($lh->{'use_external_lex_cache'}) { + local $lh->{'_external_lex_cache'}{'_AUTO'} = 1; + return $lh->maketext( $key, @args ); + } + else { + no strict 'refs'; + local ${ $lh->get_base_class() . '::Lexicon' }{'_AUTO'} = 1; + return $lh->maketext( $key, @args ); + } } ); }
Subject: [1.14 target - item #1] Support readonly lexicon hashes (patch included)
Changing title to reflect preference over obj rt
To make it easier to review and implement I've put all changes into a single small patch. This is a complete w/ Changelog, POD, tests, and version 1.13_82 -> 1.13_83 The POD describes when this feature would be necessary.
diff -ruN Locale-Maketext-1.13_82.orig/ChangeLog Locale-Maketext-1.13_82/ChangeLog --- Locale-Maketext-1.13_82.orig/ChangeLog 2009-06-23 21:29:36.000000000 -0500 +++ Locale-Maketext-1.13_82/ChangeLog 2009-07-30 12:26:19.000000000 -0500 @@ -1,5 +1,10 @@ Revision history for Perl suite Locale::Maketext +2009-08-01 Adriano Ferreira + * Development release 1.13_83 + + "Readonly" lexicon support (thanks Dan Muey) + 2009-06-23 Adriano Ferreira * Development release 1.13_82 diff -ruN Locale-Maketext-1.13_82.orig/MANIFEST Locale-Maketext-1.13_82/MANIFEST --- Locale-Maketext-1.13_82.orig/MANIFEST 2009-06-23 19:16:50.000000000 -0500 +++ Locale-Maketext-1.13_82/MANIFEST 2009-07-30 13:12:26.000000000 -0500 @@ -12,6 +12,7 @@ README t/00_load.t t/01_about_verbose.t +t/04.use_external_lex_cache.t t/10_make.t t/20_get.t t/30_local.t diff -ruN Locale-Maketext-1.13_82.orig/lib/Locale/Maketext.pm Locale-Maketext-1.13_82/lib/Locale/Maketext.pm --- Locale-Maketext-1.13_82.orig/lib/Locale/Maketext.pm 2009-06-23 21:23:47.000000000 -0500 +++ Locale-Maketext-1.13_82/lib/Locale/Maketext.pm 2009-07-30 13:07:35.000000000 -0500 @@ -10,7 +10,7 @@ BEGIN { unless(defined &DEBUG) { *DEBUG = sub () {0} } } # define the constant 'DEBUG' at compile-time -$VERSION = '1.13_82'; +$VERSION = '1.13_83'; $VERSION = eval $VERSION; @ISA = (); @@ -186,27 +186,44 @@ # Look up the value: my $value; - foreach my $h_r ( - @{ $isa_scan{ref($handle) || $handle} || $handle->_lex_refs } - ) { - DEBUG and warn "* Looking up \"$phrase\" in $h_r\n"; - if(exists $h_r->{$phrase}) { - DEBUG and warn " Found \"$phrase\" in $h_r\n"; - unless(ref($value = $h_r->{$phrase})) { - # Nonref means it's not yet compiled. Compile and replace. - $value = $h_r->{$phrase} = $handle->_compile($value); + if (exists $handle->{'_external_lex_cache'}{$phrase}) { + DEBUG and warn "* Using external lex cache version of \"$phrase\"\n"; + $value = $handle->{'_external_lex_cache'}{$phrase}; + } + else { + foreach my $h_r ( + @{ $isa_scan{ref($handle) || $handle} || $handle->_lex_refs } + ) { + DEBUG and warn "* Looking up \"$phrase\" in $h_r\n"; + if(exists $h_r->{$phrase}) { + DEBUG and warn " Found \"$phrase\" in $h_r\n"; + unless(ref($value = $h_r->{$phrase})) { + # Nonref means it's not yet compiled. Compile and replace. + if ($handle->{'use_external_lex_cache'}) { + $value = $handle->{'_external_lex_cache'}{$phrase} = $handle->_compile($value); + } + else { + $value = $h_r->{$phrase} = $handle->_compile($value); + } + } + last; } - last; - } - elsif($phrase !~ m/^_/s and $h_r->{'_AUTO'}) { - # it's an auto lex, and this is an autoable key! - DEBUG and warn " Automaking \"$phrase\" into $h_r\n"; - - $value = $h_r->{$phrase} = $handle->_compile($phrase); - last; + # extending packages need to be able to localize _AUTO and if readonly can't "local $h_r->{'_AUTO'} = 1;" + # but they can "local $handle->{'_external_lex_cache'}{'_AUTO'} = 1;" + elsif($phrase !~ m/^_/s and ($handle->{'use_external_lex_cache'} ? ( exists $handle->{'_external_lex_cache'}{'_AUTO'} ? $handle->{'_external_lex_cache'}{'_AUTO'} : $h_r->{'_AUTO'} ) : $h_r->{'_AUTO'})) { + # it's an auto lex, and this is an autoable key! + DEBUG and warn " Automaking \"$phrase\" into $h_r\n"; + if ($handle->{'use_external_lex_cache'}) { + $value = $handle->{'_external_lex_cache'}{$phrase} = $handle->_compile($phrase); + } + else { + $value = $h_r->{$phrase} = $handle->_compile($phrase); + } + last; + } + DEBUG>1 and print " Not found in $h_r, nor automakable\n"; + # else keep looking } - DEBUG>1 and warn " Not found in $h_r, nor automakable\n"; - # else keep looking } unless(defined($value)) { diff -ruN Locale-Maketext-1.13_82.orig/lib/Locale/Maketext.pod Locale-Maketext-1.13_82/lib/Locale/Maketext.pod --- Locale-Maketext-1.13_82.orig/lib/Locale/Maketext.pod 2009-06-20 14:49:46.000000000 -0500 +++ Locale-Maketext-1.13_82/lib/Locale/Maketext.pod 2009-07-30 12:40:31.000000000 -0500 @@ -937,6 +937,25 @@ arbitrarily decided to use a leading underscore as a signal to distinguish those few. +=head1 READONLY LEXICONS + +If your lexicon is a tied hash the simple act of caching the compiled value can be fatal. + +For example a L<GDBM_File> GDBM_READER tied hash will die with something like: + + gdbm store returned -1, errno 2, key "..." at ... + +All you need to do is turn on caching outside of the lexicon hash itself like so: + + sub init { + my ($lh) = @_; + ... + $lh->{'use_external_lex_cache'} = 1; + ... + } + +And then instead of storing the compiled value in the lexicon hash it will store it in $lh->{'_external_lex_cache'} + =head1 CONTROLLING LOOKUP FAILURE If you call $lh->maketext(I<key>, ...parameters...), diff -ruN Locale-Maketext-1.13_82.orig/t/04.use_external_lex_cache.t Locale-Maketext-1.13_82/t/04.use_external_lex_cache.t --- Locale-Maketext-1.13_82.orig/t/04.use_external_lex_cache.t 1969-12-31 18:00:00.000000000 -0600 +++ Locale-Maketext-1.13_82/t/04.use_external_lex_cache.t 2009-07-30 13:04:30.000000000 -0500 @@ -0,0 +1,40 @@ +use Test::More tests => 11; + +BEGIN { + chdir 't'; + unshift @INC, qw(lib ../lib); + use_ok('Locale::Maketext'); +}; + +package MyTestLocale; + +@MyTestLocale::ISA = qw(Locale::Maketext); +%MyTestLocale::Lexicon = (); +%MyTestLocale::Lexicon = (); # to avoid warnings + +package MyTestLocale::fr; + +@MyTestLocale::fr::ISA = qw(MyTestLocale); + +%MyTestLocale::fr::Lexicon = ( + '_AUTO' => 1, + 'Hello World' => 'Bonjour Monde', +); + +package main; + +my $lh = MyTestLocale->get_handle('fr'); +$lh->{'use_external_lex_cache'} = 1; +ok(exists $MyTestLocale::fr::Lexicon{'Hello World'} && !ref $MyTestLocale::fr::Lexicon{'Hello World'}, 'lex value not a ref'); + +ok($lh->maketext('Hello World') eq 'Bonjour Monde', 'renders correctly first time'); +ok(exists $lh->{'_external_lex_cache'}{'Hello World'} && ref $lh->{'_external_lex_cache'}{'Hello World'}, 'compiled into lex_cache'); +ok(exists $MyTestLocale::fr::Lexicon{'Hello World'} && !ref $MyTestLocale::fr::Lexicon{'Hello World'}, 'lex value still not a ref'); + +ok($lh->maketext('Hello World') eq 'Bonjour Monde', 'renders correctly second time time'); +ok(exists $lh->{'_external_lex_cache'}{'Hello World'} && ref $lh->{'_external_lex_cache'}{'Hello World'}, 'still compiled into lex_cache'); +ok(exists $MyTestLocale::fr::Lexicon{'Hello World'} && !ref $MyTestLocale::fr::Lexicon{'Hello World'}, 'lex value still not a ref'); + +ok($lh->maketext('This is not a key') eq 'This is not a key', '_AUTO renders correctly first time'); +ok(exists $lh->{'_external_lex_cache'}{'This is not a key'} && ref $lh->{'_external_lex_cache'}{'This is not a key'}, '_AUTO compiled into lex_cache'); +ok(!exists $MyTestLocale::fr::Lexicon{'This is not a key'}, '_AUTO lex value not added to lex');
On Thu Jul 30 14:20:05 2009, DMUEY wrote: Show quoted text
> To make it easier to review and implement I've put all changes into a > single small patch. > > This is a complete w/ Changelog, POD, tests, and version 1.13_82 ->
1.13_83 Show quoted text
> > The POD describes when this feature would be necessary.
A ready-to-upload-via-pause tarball of said patch
Download Locale-Maketext-1.13_83.tar.gz
application/x-gzip 48.3k

Message body not shown because it is not plain text.

From: toddr [...] null.net
On Thu Jul 30 14:20:05 2009, DMUEY wrote: Show quoted text
> To make it easier to review and implement I've put all changes into a > single small patch. > > This is a complete w/ Changelog, POD, tests, and version 1.13_82 ->
1.13_83 Show quoted text
> > The POD describes when this feature would be necessary.
forked perl blead on github - http://github.com/toddr/perl/tree/toddr/maketext-locale will be submitting to P5P post 5.12.0 freeze per advice from jessee http://github.com/toddr/perl/commit/52527e24a19e76942f10372430 dc63c2a7df4fd0
Accepted into blead by p5p commit ace47d680c1383b41a705467fadb2c64e7f39c71 Author: Todd Rinaldo <toddr@cpanel.net> Date: Tue Jul 6 01:28:00 2010 -0400 Locale::Maketext external cache support This patch with tests provides RO support for lexicon hashes in Locale::Maketext. This allows you to have GDBM language files owned by root which can be accessed by non-root, but not altered. If your lexicon is a tied hash the simple act of caching the compiled value can be fatal. For example a GDBM_File GDBM_READER tied hash will die with something like: gdbm store returned -1, errno 2, key "..." at ... All you need to do is turn on caching outside of the lexicon hash itself like so: sub init { my ($lh) = @_; ... $lh->{'use_external_lex_cache'} = 1; ... } And then instead of storing the compiled value in the lexicon hash it will store it in $lh->{'_external_lex_cache'} I've verified that blead is the authoritative location for Locale::Maketext source. Signed-off-by: David Golden <dagolden@cpan.org>
1.15_02 is the RC for 1.16. No issues so far. closing ticket.