Bug #83490 for Glib: options.t fails for non-utf8 locale

Wed Feb 20 17:15:47 2013 SREZIC [...] cpan.org - Ticket created

Subject:

options.t fails for non-utf8 locale

See subject. A sample test report using the "C" locale: http://www.cpantesters.org/cpan/report/8ac52868-6f76-11e2-9b43-fad0e278d1a1 Regards, Slaven

Sun Mar 03 15:26:44 2013 SREZIC [...] cpan.org - Correspondence added

On 2013-02-20 17:15:47, SREZIC wrote: Show quoted text

> See subject. A sample test report using the "C" locale: >

http://www.cpantesters.org/cpan/report/8ac52868-6f76-11e2-9b43-fad0e278d1a1 Show quoted text

>

Same problem with 1.291, it seems. Regards, Slaven

Sun Mar 03 15:26:46 2013 SREZIC [...] cpan.org - Status changed from 'new' to 'open'

Thu Apr 18 20:07:01 2013 XAOC [...] cpan.org - Taken

Thu Apr 18 20:07:49 2013 XAOC [...] cpan.org - Correspondence added

Submitted the attached patch to the GTK-Perl mailing list for comment.

Subject:

rt83490-skip_options_parse_7bit_locale.diff

diff --git a/t/options.t b/t/options.t index dc807e0..6ed0458 100644 --- a/t/options.t +++ b/t/options.t @@ -156,7 +156,9 @@ my $entries = [ } # Test that there is no double-encoding for utf8-encoded strings. - { +SKIP: { + skip("Can't test parsing of wide-byte args with 7-bit locale", 4) + if ( $ENV{"LANG"} eq "C" || $ENV{"LANG"} eq "POSIX" ); @ARGV = qw(-s â¤ â¤); $context -> parse();

Fri Apr 19 02:11:50 2013 SREZIC [...] cpan.org - Correspondence added

On 2013-04-18 20:07:49, XAOC wrote: Show quoted text

> Submitted the attached patch to the GTK-Perl mailing list for comment.

Maybe it's better to use I18N::Langinfo (in perl core since 5.8.0) for checking? After all, it's probably not only LANG which should be checked, but also LC_ALL. And there are also other values to force a 7bit locale, e.g. by using en_GB.US-ASCII. See below for some examples. BTW, are there other 7bit locales besides US-ASCII? $ env LANG=de_DE.ISO8859-1 LC_ALL= perl -MI18N::Langinfo=langinfo,CODESET -e 'warn langinfo(CODESET)' ISO8859-1 at -e line 1. $ env LANG=de_DE.ISO8859-1 LC_ALL=hr_HR.ISO8859-2 perl -MI18N::Langinfo=langinfo,CODESET -e 'warn langinfo(CODESET)' ISO8859-2 at -e line 1. $ env LANG=de_DE.ISO8859-1 LC_ALL=C perl -MI18N::Langinfo=langinfo,CODESET -e 'warn langinfo(CODESET)' US-ASCII at -e line 1. $ env LANG=de_DE.ISO8859-1 LC_ALL=POSIX perl -MI18N::Langinfo=langinfo,CODESET -e 'warn langinfo(CODESET)' US-ASCII at -e line 1. $ env LANG= LC_ALL= perl -MI18N::Langinfo=langinfo,CODESET -e 'warn langinfo(CODESET)' US-ASCII at -e line 1. $ env LANG= LC_ALL=de_DE.UTF-8 perl -MI18N::Langinfo=langinfo,CODESET -e 'warn langinfo(CODESET)' UTF-8 at -e line 1. $ env LANG= LC_ALL=en_GB.US-ASCII perl -MI18N::Langinfo=langinfo,CODESET -e 'warn langinfo(CODESET)' US-ASCII at -e line 1.

Sun Jun 23 04:22:04 2013 intrigeri [...] debian.org - Correspondence added

Subject:	[rt.cpan.org #83490] blocking upload to Debian
Date:	Sun, 23 Jun 2013 10:21:40 +0200
To:	bug-Glib [...] rt.cpan.org
From:	intrigeri <intrigeri [...] debian.org>

Hi, any news on that front? For what it's worth, this bug is blocking me from uploading the latest and greatest Glib to Debian. Cheers!

Mon Jun 24 02:48:20 2013 XAOC [...] cpan.org - Correspondence added

On Sun Jun 23 04:22:04 2013, intrigeri@debian.org wrote: Show quoted text

> Hi, > > any news on that front? > > For what it's worth, this bug is blocking me from uploading the latest > and greatest Glib to Debian. > > Cheers!

Can you try the attached patch? All of the ISO-8859 code pages are 8-bit, one byte codepages, so I also check for them as well, since the test that's failing is for multi-byte Unicode characters. I tested with the following LC_ALL/LANG combinations, as per Slaven's previous reply; LANG=de_DE.ISO8859-1 LC_ALL= LANG=de_DE.ISO8859-1 LC_ALL=hr_HR.ISO8859-2 LANG=de_DE.ISO8859-1 LC_ALL=C LANG=de_DE.ISO8859-1 LC_ALL=POSIX LANG= LC_ALL= LANG= LC_ALL=de_DE.UTF-8 LANG= LC_ALL=en_GB.US-ASCII

Subject:

rt83490-skip_options_parse_7bit_locale.23Jun2013.diff

diff --git a/t/options.t b/t/options.t index dc807e0..67fa741 100644 --- a/t/options.t +++ b/t/options.t @@ -156,7 +156,21 @@ my $entries = [ } # Test that there is no double-encoding for utf8-encoded strings. - { +SKIP: { + my $codeset; + # This eval() was taken from <https://metacpan.org/module/I18N::Langinfo>. + eval { + require I18N::Langinfo; + I18N::Langinfo->import(qw(langinfo CODESET)); + $codeset = langinfo(CODESET()); # note the () + }; + # If there was an error importing I18N::Langinfo, then the '$codeset' test + # below will be unreliable; in that case, skip this block of tests. + skip("Can't test parsing of wide-byte args; I18N::Langinfo unavailable", 4) + if ( length($@) > 0 ); + # If LC_ALL/LANG is currently set to a 7-bit locale, also skip these tests. + skip("Can't test parsing of wide-byte args with 7-bit locale", 4) + if ( $codeset eq 'US-ASCII' || $codeset =~ /ISO8859/ ); @ARGV = qw(-s â¤ â¤); $context -> parse();

Tue Jun 25 14:33:09 2013 XAOC [...] cpan.org - Correspondence added

I thought about this a bit, and decided the better way to do this is to test for a UTF-8 encoding, instead of trying to cover all of the various non-wide-character encodings, which would be considerable. Please try the attached patch and let me know if it works. Thanks, Brian You can see skipping with all of the LANG/LC_ALL settings using: LANG=de_DE.ISO8859-1 LC_ALL= \ prove -v -I blib/lib/ -Iblib/arch/ t/options.t LANG=de_DE.ISO8859-1 LC_ALL=hr_HR.ISO8859-2 \ prove -v -I blib/lib/ -Iblib/arch/ t/options.t LANG=de_DE.ISO8859-1 LC_ALL=C \ prove -v -I blib/lib/ -Iblib/arch/ t/options.t LANG=de_DE.ISO8859-1 LC_ALL=POSIX \ prove -v -I blib/lib/ -Iblib/arch/ t/options.t LANG= LC_ALL= \ prove -v -I blib/lib/ -Iblib/arch/ t/options.t LANG= LC_ALL=de_DE.UTF-8 \ prove -v -I blib/lib/ -Iblib/arch/ t/options.t LANG= LC_ALL=en_GB.US-ASCII \ prove -v -I blib/lib/ -Iblib/arch/ t/options.t LANG= LC_ALL=ISO_10646 \ prove -v -I blib/lib/ -Iblib/arch/ t/options.t

Subject:

rt83490-skip_options_non_UTF-8_locale.24Jun2013.diff

diff --git a/t/options.t b/t/options.t index dc807e0..d6fed84 100644 --- a/t/options.t +++ b/t/options.t @@ -156,7 +156,22 @@ my $entries = [ } # Test that there is no double-encoding for utf8-encoded strings. - { +SKIP: { + my $codeset; + # This eval() was taken from <https://metacpan.org/module/I18N::Langinfo>. + eval { + require I18N::Langinfo; + I18N::Langinfo->import(qw(langinfo CODESET)); + $codeset = langinfo(CODESET()); # note the () + }; + # If there was an error importing I18N::Langinfo, then the '$codeset' test + # below will be unreliable; in that case, skip this block of tests. + skip("Can't check langinfo (LC_ALL/LANG); I18N::Langinfo unavailable", 4) + if ( length($@) > 0 ); + # If LC_ALL/LANG is not some variant of UTF-8 (8-bit, wide characters), + # skip these tests. + skip("Can't test parsing of wide-byte args (non-UTF-8 locale: $codeset)", 4) + if ( $codeset !~ /UTF-8|utf8/i ); @ARGV = qw(-s â¤ â¤); $context -> parse();

Sun Jun 30 07:01:24 2013 kaffeetisch [...] gmx.de - Correspondence added

Subject:	Re: [rt.cpan.org #83490] options.t fails for non-utf8 locale
Date:	Sun, 30 Jun 2013 13:01:07 +0200
To:	bug-Glib [...] rt.cpan.org
From:	Torsten Schoenfeld <kaffeetisch [...] gmx.de>

On 25.06.2013 20:33, Brian Manning via RT wrote: Show quoted text

> I thought about this a bit, and decided the better way to do this is > to test for a UTF-8 encoding, instead of trying to cover all of the > various non-wide-character encodings, which would be considerable.

Yes, this appears to be the most robust solution. Patch looks good to me. Thanks for working on this.

Sun Jul 07 02:35:38 2013 XAOC [...] cpan.org - Correspondence added

Pushed updated fix with Kevin Ryde's suggestions in git commit ID 215711b4, and made a stable release of Glib, version 1.301, with all of the appropriate release announcements. Look for the Glib release tarball to replicate across all of the CPAN mirrors within the next 24 hours, or pick up the Sourceforge copy if you need it right away. Resolving ticket. Thanks for the original report, and sorry for the delay in getting the fix out there.

Sun Jul 07 02:35:40 2013 XAOC [...] cpan.org - Status changed from 'open' to 'resolved'

Sun Jul 07 10:54:45 2013 XAOC [...] cpan.org - Fixed in 1.301 added