Bug #82856 for libintl-perl: Tests of Locale::TextDomain fail almost everywhere

Mon Jan 21 06:55:12 2013 MITHALDU [...] cpan.org - Ticket created

Subject:

Tests of Locale::TextDomain fail almost everywhere

Heya, i don't know if you noticed, but tests of Locale::TextDomain are failing on almost every perl/system combination. Another thing i noticed while trying to figure out what happened. The changelog was previously an actual changelog file: https://metacpan.org/source/GUIDO/libintl-perl-1.20/ChangeLog Now it's just a less than helpful dump from git log: https://metacpan.org/source/GUIDO/libintl-perl-1.21/ChangeLog

Mon Jan 21 06:56:37 2013 MITHALDU [...] cpan.org - Correspondence added

Additionally it seems that your git repo is not up-to-date with the current release.

Mon Jan 21 06:56:37 2013 MITHALDU [...] cpan.org - Status changed from 'new' to 'open'

Mon Jan 21 11:03:51 2013 GUIDO [...] cpan.org - Correspondence added

Hi, On Mon Jan 21 06:55:12 2013, MITHALDU wrote: Show quoted text

> Heya, i don't know if you noticed, but tests of Locale::TextDomain are > failing on almost every perl/system combination.

Yes, I will release 1.22 this week. The problem (I think so at least) arises, when you don't have a locale de_AT installed. I should make the tests more robust and check for other suitable locales but that's a lot of work. Show quoted text

> > Another thing i noticed while trying to figure out what happened. The > changelog was previously an actual changelog file: > > https://metacpan.org/source/GUIDO/libintl-perl-1.20/ChangeLog > > Now it's just a less than helpful dump from git log: > > https://metacpan.org/source/GUIDO/libintl-perl-1.21/ChangeLog

I switched from cvs to git and I haven't found a good replacement for cvs2cl. But I'm open to suggestions.

Mon Jan 21 11:04:53 2013 GUIDO [...] cpan.org - Correspondence added

On Mon Jan 21 06:56:37 2013, MITHALDU wrote: Show quoted text

> Additionally it seems that your git repo is not up-to-date with the > current release.

Are you in the branch v1? The master branch is for 2.x.

Mon Jan 21 11:08:04 2013 MITHALDU [...] cpan.org - Correspondence added

Show quoted text

> Yes, I will release 1.22 this week.

Great. :) Show quoted text

> The problem (I think so at least) arises, when you don't have a locale > de_AT installed. I should make the tests more robust and check for > other suitable locales but that's a lot of work.

Sounds more like the test should use a hard-coded locale and not even touch the system, but i might be misunderstanding what it does. Show quoted text

> I switched from cvs to git and I haven't found a good replacement for > cvs2cl. But I'm open to suggestions.

git log has many formatting options, you could try this for a starter: git log --pretty="* %s"

Mon Jan 21 11:25:08 2013 MITHALDU [...] cpan.org - Correspondence added

Show quoted text

> Are you in the branch v1? > > The master branch is for 2.x.

Ah, i didn't realize there were multiple branches. Thanks. :)

Tue Jan 22 02:29:40 2013 GUIDO [...] cpan.org - Correspondence added

On Mon Jan 21 11:08:04 2013, MITHALDU wrote: Show quoted text

> > The problem (I think so at least) arises, when you don't have a locale > > de_AT installed. I should make the tests more robust and check for > > other suitable locales but that's a lot of work.

> > Sounds more like the test should use a hard-coded locale and not even > touch the system, but i might be misunderstanding what it does.

It is completely impossible to trigger any translation, when you don't have at least one locale installed on the system. See here: https://rt.cpan.org/Ticket/Display.html?id=81315 I should iterate over a list of possibly installed locales, pick the first one found and use that one. And if none can be found I have to skip the gettext tests altogether. Instead I currently check whether de_AT works, and use that one or skip if it is not present. Changing the strategy to a variable locale identifier requires touching every single test in a way that cannot be done with search and replace. In other words: lazyness ... Show quoted text

> > I switched from cvs to git and I haven't found a good replacement for > > cvs2cl. But I'm open to suggestions.

> > git log has many formatting options, you could try this for a starter: > > git log --pretty="* %s"

Yes, sure, but the important part - imho - would be to add the names of the files involved in the log messages and that is not trivial with git. There was some way to do it but I have to investigate again. In other words: see above ... ;)

Tue Jan 22 02:43:35 2013 GUIDO [...] cpan.org - Correspondence added

On Mon Jan 21 11:03:51 2013, GUIDO wrote: Show quoted text

> Hi, > > On Mon Jan 21 06:55:12 2013, MITHALDU wrote:

> > Heya, i don't know if you noticed, but tests of Locale::TextDomain are > > failing on almost every perl/system combination.

One more remark: This mass failure is actually a good sign. It is caused by the fix for https://rt.cpan.org/Ticket/Dis play.html?id=81315 Before that fix, it was able to make the pure Perl version return translations although no locale was installed. Now, the pure Perl version behaves as the XS version with that respect. Another downside of the fix will be that switching to the pure Perl version, when you want translations although your system lacks the required locale definitions will no longer work. This could be problematic for some users but I think that compatibility to GNU gettext is more important than these dubious backdoors. Instead, I plan to offer functions in libintl-perl 2.x that offer messsage retrieval directly from mo or po files. Then people can bypass setlocale() altogether which is usually a bad idea because you may end up mixing output based on multiple locales, or even worse using multiple character sets. But in certain situations, especially for server applications, this could work and you would not only gain performance but also thread-safety.

Tue Jan 22 05:40:35 2013 MITHALDU [...] cpan.org - Correspondence added

Show quoted text

> It is completely impossible to trigger any translation, when you don't > have at least one locale installed on the system.

Is this about retrieving translations from application-specific po files, or about getting data from the OS itself on things like dates and number formats? Show quoted text

> Instead, I plan to offer functions in libintl-perl 2.x that offer > messsage retrieval directly from mo or po files. Then people can > bypass setlocale() altogether which is usually a bad idea because you > may end up mixing output based on multiple locales, or even worse > using multiple character sets. But in certain situations, especially > for server applications, this could work and you would not only gain > performance but also thread-safety.

Honestly, that would be the best way. Maybe even as a separate dist. I looked at Locale::TextDomain as i need to localize a web application. As i'm sure you can imagine, my users don't care about the locale of my server and everyone wants to see their own language. Right now there is so much magic and implicity and globals in Locale::TextDomain, that i cannot even tell if my use-case is possible in a sane manner and as such cannot use it with a clean conscience and will need to reimplement its syntax in a clean and simple module. Show quoted text

> Yes, sure, but the important part - imho - would be to add the names > of the files involved in the log messages and that is not trivial > with git.

Sure: git log --pretty="format:* %s" --name-only There's plenty other options like shortstat, numstat, stat, name-status, etc. Just pick whatever you like. Personally though, the point was rather: Perl dists should have a concise, useful and human-written summary of changes between releases. I looked at the changelogs in hopes of figuring out whether the last release was for bugfixes, or further development; to find out the significant changes. A prettier dump of git log won't tell me that either. Also see: http://changes.cpanhq.org

Tue Jan 22 07:23:20 2013 GUIDO [...] cpan.org - Correspondence added

On Tue Jan 22 05:40:35 2013, MITHALDU wrote: Show quoted text

> > It is completely impossible to trigger any translation, when you

> don't

> > have at least one locale installed on the system.

> > Is this about retrieving translations from application-specific po > files, or about > getting data from the OS itself on things like dates and number > formats?

All gettext() functions will return the original msgid, unless you could successfully switch to a locale other than C/POSIX. Show quoted text

> > Instead, I plan to offer functions in libintl-perl 2.x that offer > > messsage retrieval directly from mo or po files. Then people can > > bypass setlocale() altogether which is usually a bad idea because

> you

> > may end up mixing output based on multiple locales, or even worse > > using multiple character sets. But in certain situations,

> especially

> > for server applications, this could work and you would not only gain > > performance but also thread-safety.

> > Honestly, that would be the best way. Maybe even as a separate dist. I > looked at > Locale::TextDomain as i need to localize a web application. As i'm > sure you can imagine, > my users don't care about the locale of my server and everyone wants > to see their own > language. Right now there is so much magic and implicity and globals > in > Locale::TextDomain, that i cannot even tell if my use-case is possible > in a sane manner > and as such cannot use it with a clean conscience and will need to > reimplement its > syntax in a clean and simple module.

The globals in Locale::TextDomain and Locale::gettext_pp are caches, things that are safe to remember, even under mod_perl. If you really want to retrieve translations directly from mo files that is quite easy. Just look at the sources of gettext_pp.pm. I could even extend Locale::Messages::select_package() so that it excepts other package identifiers than gettext_pp or gettext_xs. That way you can roll your own message retrieval backend (for example derived from gettext_pp) that bypasses all setlocale() stuff. The problem arises, when you interpolate OS output into your messages (or the other way round). Even in a web application you may want to interpolate $! (stringified) or the output of POSIX::strftime(). That can lead to issues. The OS locale could be set to fr_FR.utf-8 while you get messages for ru_RU.koi8-r. Show quoted text

> Personally though, the point was rather: Perl dists should have a > concise, useful and > human-written summary of changes between releases. I looked at the > changelogs in hopes > of figuring out whether the last release was for bugfixes, or further > development; to > find out the significant changes. A prettier dump of git log won't > tell me that either.

But the file NEWS does. For traditional reasons, libintl-perl rather follows the GNU conventions for filenames, than the Perl ones.

Wed Jan 23 08:16:58 2013 MITHALDU [...] cpan.org - Correspondence added

On Tue Jan 22 13:23:20 2013, GUIDO wrote: Show quoted text

> The problem arises, when you interpolate OS output into your messages > (or the other way round). Even in a web application you may want to > interpolate $! (stringified) or the output of POSIX::strftime(). That > can lead to issues. The OS locale could be set to fr_FR.utf-8 while

you Show quoted text

> get messages for ru_RU.koi8-r.

This bit i didn't understand, so thanks for explaining. Show quoted text

> The globals in Locale::TextDomain and Locale::gettext_pp are caches, > things that are safe to remember, even under mod_perl.

Ah, that was not clear from the way they are named. Also, how would i go about clearing those caches? Show quoted text

> If you really want to retrieve translations directly from mo files

that Show quoted text

> is quite easy. Just look at the sources of gettext_pp.pm. I could

even Show quoted text

> extend Locale::Messages::select_package() so that it excepts other > package identifiers than gettext_pp or gettext_xs. That way you can > roll your own message retrieval backend (for example derived from > gettext_pp) that bypasses all setlocale() stuff.

And another thing i didn't realize: L::TD uses mo files to get data instead of reading from the po files. Since i'm only starting to learn how gettext operates this wasn't clear to me. Show quoted text

> But the file NEWS does. For traditional reasons, libintl-perl rather > follows the GNU conventions for filenames, than the Perl ones.

That's very confusing for perl developers though, since sites like metacpan and search.cpan.org look for files labeled Changes or ChangeLog to automatically link them. Maybe add a message at the top of that file to point out the real changelog file?

Wed Jan 23 09:13:24 2013 GUIDO [...] cpan.org - Correspondence added

On Wed Jan 23 08:16:58 2013, MITHALDU wrote: Show quoted text

> On Tue Jan 22 13:23:20 2013, GUIDO wrote:

> > The globals in Locale::TextDomain and Locale::gettext_pp are caches, > > things that are safe to remember, even under mod_perl.

> > Ah, that was not clear from the way they are named. Also, how would i > go about clearing those caches?

You cannot unless you fiddle with the internals. We are talking about mod_perl safetey here, right? With that respect, one of the caches is slightly dangerous. For performance reasons, the module keeps a cache of directories it could not open. This cache - like all others - will survive HTTP requests in mod_perl. Not perfect but I think it is acceptable. The other things that Locale::gettext_pp "remembers" are the currently set textdomain, textdomain bindings, and so on. However, the XS version behaves the same and there is no way to reset it. The only thing you can do is properly initialize I18N at the beginning of every request. In other words, call setlocale(), textdomain(), bindtextdomain(), bind_textdomain_codeset(), and bind_textdomain_filters() when you start processing a request. Even then you have to keep in mind that libintl-perl is not threadsafe, because GNU gettext is not, because setlocale() is not! Therefore you cannot use a threaded mpm with mod_perl in Apache. Show quoted text

> > But the file NEWS does. For traditional reasons, libintl-perl rather > > follows the GNU conventions for filenames, than the Perl ones.

> > That's very confusing for perl developers though, since sites like > metacpan and search.cpan.org look for files labeled Changes or > ChangeLog to automatically link them. Maybe add a message at the top of > that file to point out the real changelog file?

My bad ... the file displayed in CPAN is ReleaseNotes (a copy of NEWS). So you see both ReleaseNotes and ChangeLog. The current way it is done was recommended by Graham Barr to me and he runs search.cpan.org.

Wed Jan 23 09:27:58 2013 MITHALDU [...] cpan.org - Correspondence added

I should've put the question about the caches more clearly. :) The case i have in mind is this: My app runs as fcgi, so thread-safety is not very important to me, nor mod_perl. My concern is that i wish to allow our translators to easily verify their translations in the website itself. This means that i need to be able to check whether a .mo file is of a newer date than the last time i a translation was requested, and to then force a reload. Right now it sounds like this is impossible without a restart of the process. Optimally i'd be able to instantiate L::TD objects which contain their own caches that can be modified via methods. The API inside packages would remain unchanged, as i could simply export closures on the object methods. Show quoted text

> My bad ... the file displayed in CPAN is ReleaseNotes (a copy of NEWS). > So you see both ReleaseNotes and ChangeLog. The current way it is done > was recommended by Graham Barr to me and he runs search.cpan.org.

I see. In that case i'll submit a bug report to metacpan to see if they can have their change file detector check for releasenotes and news first. On that note: Your news file is actually different from the releasenotes file in the last release.

Wed Jan 23 10:33:38 2013 GUIDO [...] cpan.org - Correspondence added

On Wed Jan 23 09:27:58 2013, MITHALDU wrote: Show quoted text

> Optimally i'd be able to instantiate L::TD objects which contain their > own caches that can be modified via methods. The > API inside packages would remain unchanged, as i could simply export > closures on the object methods.

Please don't forget that Locale::gettext_pp aims to be 100 % compatible to Locale::gettext_xs and Locale::gettext_xs is just a thin wrapper around the gettext functions in your libc or libintl. And since the underlying C code is not reentrant, the Perl bindings are not reentrant. What you can do is enforce the use of the pure Perl version of the library: Locale::Messages->select_package ('gettext_pp'); And then, when you see that a MO (not PO!) file has changed, force recompilation of the Perl module, so that all variables are thrown away: delete $INC{'Locale/gettext_pp.pm'}; require Locale::gettext_pp; Maybe, you also have re-import the symbols: Locale::gettext_pp->import(':libintl_h', 'libintl_h'); Not sure about that last one. Show quoted text

> > My bad ... the file displayed in CPAN is ReleaseNotes (a copy of

> NEWS).

> > So you see both ReleaseNotes and ChangeLog. The current way it is

> done

> > was recommended by Graham Barr to me and he runs search.cpan.org.

> > I see. In that case i'll submit a bug report to metacpan to see if > they can have their change file detector check for > releasenotes and news first. On that note: Your news file is actually > different from the releasenotes file in the last > release.

Hm, yes. The Makefile obviously has a flaw.

Thu Jan 24 10:41:37 2013 GUIDO [...] cpan.org - Correspondence added

Should be fixed in 1.22.

Thu Jan 24 10:41:38 2013 GUIDO [...] cpan.org - Status changed from 'open' to 'resolved'