Skip Menu |

This queue is for tickets about the libintl-perl CPAN distribution.

Report information
The Basics
Id: 37763
Status: resolved
Priority: 0/
Queue: libintl-perl

People
Owner: GUIDO [...] cpan.org
Requestors: svysh [...] pn.sinp.msu.ru
Cc:
AdminCc:

Bug Information
Severity: Normal
Broken in: 1.16
Fixed in: (no value)



Subject: Locale::gettext_xs::gettext fails when called via Apache2.2 with mod_perl2
libintl-perl-1.16 perl v5.8.8 freebsd 6.3 Locale::gettext_xs::gettext works ok from regular perl script, and from under Apache1.3 with mod_perl, and from under Apache2.2 with mod_cgi. It does not work from under Apache2.2 with mod_perl2. "Does not work" means: it returns supplied argument without translating it. Very similar to the explicit bug (reported as #37762) in Locale::Messages::select_package ?
I successfully use Locale::gettext_xs with Apache 2.2 and mod_perl2, and since gettext_xs is a mere wrapper around GNU gettext (runtime), it's very unlikely that the behavior you experience is due to a bug in libintl-perl. You can easily test that yourself: Install Locale::gettext (the C only version), and make sure that it can find your .mo file(s). You can verify the latter, by running a simple test script on the command line. If it is running, try it under mod_perl2. If Locale::gettext has the same problem as Locale::gettext_xs it is a problem of your installation. How do you select the locale in your application? Try to hardcode the locale directly with POSIX::setlocale: POSIX::setlocale LC_ALL => 'bg_BG.utf8'; Make sure that the argument you pass is valid, by checking "locale -a". Does the pure Perl version (Locale::gettext_pp) work? If it does, switch to it until you fixed the issue with: Locale::Messages->select_package ('gettext_pp'); Don't worry too much about performance. Unless you are constantly calling __something in loops, the pure Perl version can really compete with the XS version.
Hi Guido, thanks for your prompt reply. I work with Sergei on the OpenXPKI project (an open source trustcenter software), which is where this bug occurs. On Fri Jul 18 02:31:14 2008, GUIDO wrote: Show quoted text
> I successfully use Locale::gettext_xs with Apache 2.2 and mod_perl2, and > since gettext_xs is a mere wrapper around GNU gettext (runtime), it's > very unlikely that the behavior you experience is due to a bug in > libintl-perl.
On which system do you use this? We have successfully used it under both Debian and Solaris 10 with Apache 2.2 and mod_perl2, but it fails both under FreeBSD (on Sergei's machine) and Mac OS X (on my machine). Our (internal) bug report is here for your reference: http://sourceforge.net/tracker/index.php?func=detail&aid=1806325&group_id=150124&atid=776757 Show quoted text
> How do you select the locale in your application? Try to hardcode the > locale directly with POSIX::setlocale: > > POSIX::setlocale LC_ALL => 'bg_BG.utf8';
We are doing (POSIX::)setlocale for LC_MESSAGES and LC_TIME, see http://openxpki.svn.sf.net/viewvc/openxpki/trunk/perl-modules/core/trunk/OpenXPKI/i18n.pm?revision=865&view=markup Show quoted text
> Make sure that the argument you pass is valid, by checking "locale -a".
Yes, I've verified that. Show quoted text
> Does the pure Perl version (Locale::gettext_pp) work? If it does, switch > to it until you fixed the issue with: > > Locale::Messages->select_package ('gettext_pp');
Sort of. Sometimes it returns the correct result, sometimes just the identifier - weird ... - !? Show quoted text
> Don't worry too much about performance. Unless you are constantly > calling __something in loops, the pure Perl version can really compete > with the XS version.
OK, that sounds promising at least, now we only need to get the PP version running ... I've attached a mod_perl2 handler that we've written while debugging this problem to rule out Mason as the culprit, I've attached that for your reference. From there you can also see that when running gettext using the command line, everything works fine, so I think mod_perl2 is involved somehow ... Thanks again for your help, Best regards, Alex
package OpenXPKI::TestModPerlHandler; use strict; use warnings; use Locale::Messages qw (:locale_h :libintl_h nl_putenv bind_textdomain_filter select_package gettext); use Locale::gettext_pp; use POSIX qw (setlocale); use Encode; use Data::Dumper; sub handler { my $r = shift; select_package("gettext_pp"); my $language = 'de_DE'; my $locale_prefix = '/Users/klink/usr/local/share/locale'; my $loc = "${language}.UTF-8"; setlocale('LC_MESSAGES', $loc); setlocale('LC_TIME', $loc); setlocale('LC_ALL', $loc); Locale::Messages::nl_putenv("LC_MESSAGES=$loc"); Locale::Messages::nl_putenv("LC_TIME=$loc"); Locale::Messages::nl_putenv("LC_ALL=$loc"); Locale::Messages::nl_putenv("LANG=$loc"); Locale::Messages::nl_putenv("LANGUAGE=$loc"); my $eall = $ENV{'LC_ALL'}; my $emessages = $ENV{'LC_MESSAGES'}; my $elang = $ENV{'LANG'}; my $elanguage = $ENV{'LANGUAGE'}; Locale::Messages::textdomain("openxpki"); Locale::Messages::bindtextdomain("openxpki", $locale_prefix); Locale::Messages::bind_textdomain_codeset("openxpki", "UTF-8"); ## This line fails with Apache-2.x on FreeBSD my $data = gettext("I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE"); my $data_pp = Locale::gettext_pp::gettext('I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE'); ## The next line is an example of suggested workaround my $data_cmd = `export LANG=de_DE.UTF-8 && export LANGUAGE=de_DE.UTF-8 && /opt/local/bin/gettext -d openxpki I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE`; Encode::_utf8_on($data); print "Perl gettext: $data\nCommandline: $data\nPerl gettext_pp: $data_pp\n"; $Data::Dumper::Sortkeys = 1; print Dumper \%ENV; return 0; } 1;
From: svysh [...] pn.sinp.msu.ru
Hi, Let me clarify. For me (freebsd 7.0 or 6.3, Apache2.2, modperl2) Locale::gettext_pp version works just fine. Does it mean that locale settings (from the point of view of xs version) could be still incorrect? What could be the reason for pp working, while xs not working? Regards, Sergei
On Fri Jul 18 08:12:34 2008, svysh wrote: Show quoted text
> Let me clarify. > > For me (freebsd 7.0 or 6.3, Apache2.2, modperl2) > Locale::gettext_pp version works just fine.
Does that mean with the TestModPerlHandler, you get consistent results of what you would expect? For the record, here is Guido's reply on the OpenXPKI tracker and my followup: rt.cpan.org has problems, so I reply here. Please check the return value from setlocale. My suspicion is that it fails. One possible reason is that on BSDish systems locale names are case-sensitive, while they are case-insensitive in the GNU implementation. In general, since locale names are not portable at all, you usually have to do a little guessing, in order to find a match. In fact, you have to do a lot of guessing ... Please check out the CVS version of libintl-perl, at :pserver:anonymous@guido-flohr.net:/home/cvsroot/perl module "libintl-perl". Use an emtpy password. You will find a file Locale/Util.pm which contains some routines that do such guesses, maybe that helps. If you use it in your project, please rename it, in order to avoid future conflicts. Regards, Guido Flohr ----- Guido, thanks for your continued support - the next time I'm in Cologne, I'll buy you a beer :-) Code for checking the result of setlocale has been checked in with r1287. At least on Mac OS X, it does not throw any exceptions, though. In the end, we have a fixed list of ['en_GB.UTF-8', 'de_DE.UTF-8', 'ru_RU.UTF-8'] values that we use with setlocale, so I doubt that this is the problem ...
From: svysh [...] pn.sinp.msu.ru
Hi Guido, To be more specific, attached please find by mod_perl2 handler and its output. See how pp version works while xs fails. ("I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE" is a tag, and "Zertifikat #__SERIAL__" is a correct translation) Do you think this could be wrong locale settings? Regards, Sergei Птн. Июл. 18 08:12:34 2008, svysh писал: Show quoted text
> Hi, > > Let me clarify. > > For me (freebsd 7.0 or 6.3, Apache2.2, modperl2) > Locale::gettext_pp version works just fine. > > Does it mean that locale settings (from the point of view of xs version) > could be still incorrect? > > What could be the reason for pp working, while xs not working? > > Regards, Sergei
Download TestModPerlHandler_pp.zip
application/x-zip-compressed 1.2k

Message body not shown because it is not plain text.

I modified the handler a little bit, and the output looks like this here: Did setlocale succeed? LC_MESSAGES: de_DE.UTF-8 LC_TIME: de_DE.UTF-8 LC_ALL: de_DE.UTF-8 Today: Montag Perl gettext from Messages: I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE Pure Perl gettext: I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE Command_line: I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE XS gettext: I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE Command_line: I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE The important part is that the calls to setlocale() all succeed. You can also see that changing LC_TIME also worked, because the day of the week is output in German. It is no wonder that nothing else gets translated, because I don't have your mo files, but that isn't important. I think I have to clarify the meaning of the environment for the GNU implementation of gettext (and libintl-perl, which is mostly compatible to GNU gettext). The file ABOUT-NLS (see for example http://cvs.savannah.gnu.org/viewvc/gettext/gettext-tools/ABOUT-NLS?revision=1.12&root=gettext&view=markup, please read part 1.3 "Using this package") recommends to users (users! not programs!) to set the LC_* variables to valid locale identifiers on their system. The only reason for that is that an internationalized program should begin like this: setlocale (LC_ALL, ""); But this is pure convention, and especially server applications like yours will usually violate that convention on purpose. The reason for that convention is that a POSIX compliant setlocale implementation in the libc will interpret the empty (empty! not undefined!) second argument so, that it will inspect the various LC_* environment variables for setting the locale. There are two more environment variables, LANG and LANGUAGE which have a special meaning. LANG defines the user's preferred language (language is a LL_CC combination with LL being the 2-letter language code and CC the optional two-letter country-code). LANG is however ignored, if the locale is set to anything else but "C" or "POSIX". The environment variable LANGUAGE should contain a colon separated list of languages, for example "de:bg_BG". A trailing charset declaration for a language, for example "de.utf-8" is gracefully stripped off and ignored. You can force a certain charset by setting the environment variable "OUTPUT_CHARSET" to a valid charset definition. BTW, if you set LANGUAGE you should still set LANG, not for gettext but for other system components, see ABOUT-NLS. So, how does gettext_pp decide on how to search for a messages catalog, say $locale_prefix/de_DE/LC_MESSAGES? If the environment variable LANGUAGE is set, then it is split at colons (:) into a list of language identifiers to try, otherwise it calls setlocale() with an undefined second argument. The return value is the current locale setting (for the first argument category). If the return value matches the expected format (LL_CC@region.charset, with everything but LL being optional), this value is taken. What happens on the BSD systems, where things don't work as expected? I can only guess, but my guess is that your calls to setlocale() fail (see my modified handler for a check for that). If you use gettext_pp, these failures are healed, because you override everything with the environment variable LANGUAGE. The behavior of GNU gettext is the same (I hope so at least), but your system's gettext may of course be implemented slightly different. So, if you want to rely on the behavior, that LANGUAGE overrides everything else, you should force the usage of gettext_pp. Is the day of the week output in German on your test system? If not, then it is also no wonder that gettext fails to translate the strings, becauses it uses more or less the same mechanism. Remember that you actually do a (legitimate) abuse of gettext, by hardcoding locale identifiers into your code. The idea of gettext is that the identifiers are set in the user's environment, because their format is completely system-dependent. On Windows a German locale may look like "German_Germany.cp1252" instead of "de_DE.utf-8". On some systems case matters, on others it doesn't. The exact spelling of charset identifiers is also system-dependent ("utf8", "utf-8", "UTF-8", ...). So, even if you know the language, the country and the charset, you still have to take wild guesses at a recognized format, and in the end you can still have a failure, if the desired locale is not installed at all on the system. See the file Locale/Util.pm that I mentioned in my posting on your sourceforge bugtracking system. The environment variables LANGUAGE and OUTPUT_CHARSET (if honored by your gettext) are a way out of that dilemma, because they interface directly with gettext, so that setlocale() is bypassed when determining the desired locale. Final advice, please s/Locale::gettext_pp::(.*)/Locale::gettext::$1/ for a test. You have to install Locale::gettext from CPAN for that. If Locale::gettext::gettext() shows the same behavior as gettext_xs from libintl-perl you can clearly see that it is not a problem of libintl-perl, but that your system's gettext simply does not work as you expect.
package TestModPerlHandler_pp; use strict; use warnings; use Locale::Messages qw (:locale_h :libintl_h nl_putenv bind_textdomain_filter select_package gettext); use POSIX qw (setlocale); use Encode; use Locale::gettext_pp; sub handler { my $r = shift; $r->content_type ("text/html; charset=utf-8"); # Guido: Workaround for a bug in libintl-perl 1.16: Call select_package() # as a class method. This will work with future versions, too. # See http://rt.cpan.org/Ticket/Display.html?id=37762 for details. my $res = Locale::Messages->select_package("gettext_pp"); print "<p>Will call module: $res</p>"; my $language = 'de_DE'; my $locale_prefix = '/usr/local/share/locale'; my $loc = "${language}.UTF-8"; # Guido: You must _always_ check the return value of setlocale(), # when you try to change the locale. In nine out of ten cases # the call fails. my $lc_messages = POSIX::setlocale(&POSIX::LC_MESSAGES, $loc); my $lc_time = POSIX::setlocale(&POSIX::LC_TIME, $loc); my $today = POSIX::strftime ("%A", localtime); my $lc_all = POSIX::setlocale(&POSIX::LC_ALL, $loc); $r->print (<<EOF); <pre> Did setlocale succeed? LC_MESSAGES: $lc_messages LC_TIME: $lc_time LC_ALL: $lc_all Today: $today </pre> EOF if (0) { # Guido: This entire block of code is useless, because the environment is # irrelevant here. The environment is _only_ used, when you call # setlocale() with an empty (empty! not undefined) second argument. # In that case, the locale is set according to the corresponding # environment variable, for example: # # nl_putenv("LC_MESSAGES=bg_BG.utf8"); # setlocale (LC_MESSAGES, ''); # # In other words: Modifying the environment is too late here, and # besides it is not needed. You always pass a second argument to # setlocale, so there is no need for changing the environment. # # There is one exception to the above: The environment variable # LANGUAGE overrides all locales settings set with setlocale(), # at least in gettext_pp. Furthermore, its format differs from # the other envariables, in that it is a colon separated list of # languages (languages! not locale identifiers). See the comment # on it in ABOUT-NLS that comes with all internationalized software # packages. Locale::Messages::nl_putenv("LC_MESSAGES=$loc"); Locale::Messages::nl_putenv("LC_TIME=$loc"); Locale::Messages::nl_putenv("LC_ALL=$loc"); Locale::Messages::nl_putenv("LANG=$loc"); Locale::Messages::nl_putenv("LANGUAGE=$loc"); my $eall = $ENV{'LC_ALL'}; my $emessages = $ENV{'LC_MESSAGES'}; my $elang = $ENV{'LANG'}; my $elanguage = $ENV{'LANGUAGE'}; print "<p>Before script works</p>"; print "<p>LC_ALL: $eall</p>"; print "<p>LC_MESSAGES: $emessages</p>"; print "<p>LANG: $elang</p>"; print "<p>LANGUAGE: $elanguage</p>"; } Locale::Messages::textdomain("openxpki"); Locale::Messages::bindtextdomain("openxpki", $locale_prefix); Locale::Messages::bind_textdomain_codeset("openxpki", "UTF-8"); ## This line calls gettext_pp via Messages::gettext and works ok with Apache-2.x on FreeBSD my $data = Locale::Messages::gettext("I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE"); Encode::_utf8_on($data); $r->print ("Perl gettext from Messages: $data<br/>"); Locale::gettext_pp::textdomain("openxpki"); Locale::gettext_pp::bindtextdomain("openxpki", $locale_prefix); Locale::gettext_pp::bind_textdomain_codeset("openxpki", "UTF-8"); ## This line calls explicitly gettext_pp and works ok on FreeBSD Apache-2.x and mod_perl2 my $data2 = Locale::gettext_pp::gettext("I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE"); my $data3 = Locale::gettext_xs::gettext("I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE"); ## The next line calls gettext from CL my $data_cmd = `export LANG=de_DE.UTF-8 && export LANGUAGE=de_DE.UTF-8 && /usr/local/bin/gettext -d openxpki I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE`; # Guido: If you want the utf-8 flag (brrrr) set on output from libintl-perl # you should use an output filter: use Locale::Messages qw (turn_utf_8_on bind_textdomain_filter); bind_textdomain_filter openxpki => \&turn_utf_8_on; if (0) { Encode::_utf8_on($data2); Encode::_utf8_on($data3); } $r->print ("Pure Perl gettext: $data2<br/>Command_line: $data_cmd <br/>"); $r->print ("XS gettext: $data3<br/>Command_line: $data_cmd <br/>"); # Guido: See above: The environment is irrelevant. #print "<p>After script worked</p>"; #print "<p>LC_ALL: $eall</p>"; #print "<p>LC_MESSAGES: $emessages</p>"; #print "<p>LANG: $elang</p>"; #print "<p>LANGUAGE: $elanguage</p>"; return 0; } 1;
From: svysh [...] pn.sinp.msu.ru
Hi Guido, Thank you so much for your detailed message. I have modified your version of the script to include Locale::gettext() call. Its output: ====================================================== Will call module: gettext_pp Did setlocale succeed? LC_MESSAGES: de_DE.UTF-8 LC_TIME: de_DE.UTF-8 LC_ALL: de_DE.UTF-8 Today: Montag Perl gettext from Messages: Zertifikat #__SERIAL__ Pure Perl gettext: Zertifikat #__SERIAL__ Command_line: Zertifikat #__SERIAL__ XS gettext: I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE Locale::gettext(): I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE ====================================================== Seems that locale is set correctly. Also it follows advice of the system utility "/usr/bin/locale -a", which calls it exactly as "de_DE.UTF-8" in freebsd. As you see, both Locale::gettext() and _xs versions do not work. Using _pp version is a nice workaround, but it seems that in our particular case (of openxpki) its use alone does not resolve all problems. The question is not who is to blame, the question is how to set (and check?) locale in a correct and dedactic way. So that BSDs are not mistreated. Thank you very much for your help again. Regard, Sergei
Zdrasti, Sergei, if Locale::gettext shows the same behavior as Locale::gettext_xs, it is clear that this is not a problem with libintl-perl but with the underlying C library. The XS versions of libintl-perl and Locale::gettext are mere wrappers around the C functions. Does FreeBSD have a tool similar to strace() for Linux? If there is such a tool, you could write a mini Perl script that translates the message in question, and call it from withing mod_perl with strace (you name it for FreeBSD). From the files it is trying to open, you will probably be able to find the error. If there is no equivalent to strace(), I am clueless now. You could only recompile the library that implements gettext() and friends, and insert some debugging output. I may get help from a FreeBSD user tomorrow. If we can reproduce the problem and find a fix, I'll let you know. Mnogo pozdravi ot Bylgariq, Guido
On Fri Jul 18 02:31:14 2008, GUIDO wrote: Show quoted text
> POSIX::setlocale LC_ALL => 'bg_BG.utf8';
Ouch! This is a very common bug, and I have done it myself: Using the fat comma (=>) will cause the left-hand side to be interpreted as a string. But you must use the numerical constant: POSIX::setlocale LC_ALL, 'bg_BG.utf8'; This is just for the records, and has nothing to do with the bug.
By the way, does the problem only occur when running under mod_perl2 or also in CGI scripts?
From: svysh [...] pn.sinp.msu.ru
Hi Guido, Thank you very much for your concern. As mentioned in the very first message: Locale::gettext_xs::gettext works ok from regular perl script, and from under Apache1.3 with mod_perl, and from under Apache2.2 with mod_cgi. It does not work from under Apache2.2 with mod_perl2. "Does not work" means: it returns supplied argument without translating it. Regards, Sergei Пнд. Июл. 21 17:35:04 2008, GUIDO писал: Show quoted text
> By the way, does the problem only occur when running under mod_perl2
or Show quoted text
> also in CGI scripts?
From: svysh [...] pn.sinp.msu.ru
Пнд. Июл. 21 17:18:05 2008, GUIDO писал: Show quoted text
> Does FreeBSD have a tool similar to strace() for Linux? > you could write a mini Perl script that translates the message > in question, and call it from withing mod_perl with strace
Hi Guido, Yes, in freebsd we have classical strace, truss, and also Devel::STrace from CPAN, but I do not see much from the output. Could you possibly hint about which mini Perl script do you have in mind and what do you want to see from it? Regards, Sergei
The Perl script should look like this: use strict; use Locale::gettext_xs qw (textdomain bindtextdomain gettext); use POSIX qw (setlocale LC_ALL); textdomain 'openxpki'; bindtextdomain openxpki => '/usr/local/share/locale'; my $translated = gettext "I18N_OPENXPKI_CLIENT_HTML_MASON_API_CERT_INFO_TITLE"; print $translated; __END__ When you trace it, you can maybe see from the various open() calls what goes wrong.
http://www.freebsd.org/cgi/man.cgi?query=setlocale&apropos=0&sektion=0&manpath=FreeBSD+4.5-RELEASE&format=html#end Could that be the reason? See the section "BUGS". Until this is fixed: Could you simply set the environment variable "LANGUAGE" to the desired language, at least for FreeBSD? That should fix the problem.
From: svysh [...] pn.sinp.msu.ru
Hi Guido, Срд. Июл. 23 08:52:53 2008, GUIDO писал: Show quoted text
man.cgi?query=setlocale&apropos=0&sektion=0&manpath=FreeBSD+4.5- Show quoted text
> RELEASE&format=html#end > > Could that be the reason? See the section "BUGS".
FreeBSD-4.5 which you quote is many years old and out of support. I was reporting problems with modern versions 6.3 and 7.0 of FreeBSD. In these version those BUGS are absent: http://www.freebsd.org/cgi/ man.cgi?query=setlocale&apropos=0&sektion=0&manpath=FreeBSD+6.3- stable&format=html Show quoted text
> > Until this is fixed: Could you simply set the environment variable > "LANGUAGE" to the desired language, at least for FreeBSD? That should > fix the problem.
I will try. Thank you very much anyway. Regards, Sergei
From: svysh [...] pn.sinp.msu.ru
Show quoted text
> Could you simply set the environment variable > "LANGUAGE" to the desired language, at least for FreeBSD? That should > fix the problem.
Sorry, I did not catch the point in my previous answer. In the very first version of my TestModPerlHandler_pp.pm sent to you, environment variable "LANGUAGE" (and "LANG" too) was set just in case, along with all other imaginable ways to explain to everybody, which language to use. And _xs version did not work. Maybe you mean something else?
$LANGUAGE: Okay, when you have tried that already, it doesn't help. I think the only way to track town the problem is that you somehow trace the system calls and check which directories resp. files are checked by the XS version.
Seems to be outdated. Please reopen, if this is still an issue.