Bug #122775 for Hash-SharedMem: Test failures with non-English locale and perl 5.27.2

Sat Aug 12 02:41:55 2017 SREZIC [...] cpan.org - Ticket created

CC:	ANDK [...] cpan.org
Subject:	Test failures with non-English locale and perl 5.27.2

With a German locale (LC_ALL=de_DE.UTF-8) and perl 5.27.2 I see the following failures: ... # Failed test at t/bad_dir.t line 20. # 'can't open shared hash /var/tmp/cpansmoker-1023/2017081121/qS_PqUtiuG/t0: Datei oder Verzeichnis nicht gefunden at t/bad_dir.t line 19. # ' # doesn't match '(?^:\Acan't open shared hash \/var\/tmp\/cpansmoker\-1023\/2017081121\/qS_PqUtiuG/t0: No\ such\ file\ or\ directory at )' # Looks like you failed 1 test of 7. t/bad_dir.t ........ Dubious, test returned 1 (wstat 256, 0x100) Failed 1/7 subtests # Failed test at t/bad_master.t line 58. # 'can't open shared hash /var/tmp/cpansmoker-1023/2017081121/LjUAs9Tob4/t0: Ist ein Verzeichnis at t/bad_master.t line 57. # ' # doesn't match '(?^x:\Acan't\ open\ shared\ hash\ \/var\/tmp\/cpansmoker\-1023\/2017081121\/LjUAs9Tob4/t0: # \ (?:Is\ a\ directory|not\ a\ shared\ hash)\ at\ )' # Failed test at t/bad_master.t line 61. # 'can't open shared hash /var/tmp/cpansmoker-1023/2017081121/LjUAs9Tob4/t0: Ist ein Verzeichnis at t/bad_master.t line 60. # ' # doesn't match '(?^x:\Acan't\ open\ shared\ hash\ \/var\/tmp\/cpansmoker\-1023\/2017081121\/LjUAs9Tob4/t0: # \ (?:Is\ a\ directory|not\ a\ shared\ hash)\ at\ )' # Looks like you failed 2 tests of 28. t/bad_master.t ..... Dubious, test returned 2 (wstat 512, 0x200) Failed 2/28 subtests ... (etc.) ... This does not happen with an English locale, or perl 5.27.1 or earlier. Quickly scanning through perldelta I don't see an entry which would explain the change of behavior. @Andreas: a bisect candidate?

Sat Aug 12 07:13:44 2017 andreas.koenig.7os6VVqR [...] franz.ak.mind.de - Correspondence added

CC:	ANDK [...] cpan.org
Subject:	Re: [rt.cpan.org #122775] Test failures with non-English locale and perl 5.27.2
Date:	Sat, 12 Aug 2017 12:33:22 +0200
To:	"Slaven_Rezic via RT" <bug-Hash-SharedMem [...] rt.cpan.org>
From:	Andreas Koenig <andreas.koenig.7os6VVqR [...] franz.ak.mind.de>

Show quoted text

>>>>> On Sat, 12 Aug 2017 02:41:56 -0400, "Slaven_Rezic via RT" <bug-Hash-SharedMem@rt.cpan.org> said:

Show quoted text

> @Andreas: a bisect candidate?

Bisect leads to: : ec268cc8df7c7a90811a099d422eef6a31bf9f8b is the first bad commit : commit ec268cc8df7c7a90811a099d422eef6a31bf9f8b : Author: Karl Williamson <khw@cpan.org> : Date: Sat Jul 15 12:36:54 2017 -0600 : : embed.fnc: Fix declaration of my_strerror() BTW, while at it, I discovered that it always fails with the russian locale ru_RU.utf8 also for 5.27.1, 5.26.0, and 5.24.2 (did not test more). -- andreas

Sat Aug 12 07:14:16 2017 The RT System itself - Status changed from 'new' to 'open'

Sat Aug 12 07:56:42 2017 zefram [...] fysh.org - Correspondence added

Subject:	Re: [rt.cpan.org #122775] Test failures with non-English locale and perl 5.27.2
Date:	Sat, 12 Aug 2017 11:43:59 +0100
To:	Slaven_Rezic via RT <bug-Hash-SharedMem [...] rt.cpan.org>
From:	Zefram <zefram [...] fysh.org>

Slaven_Rezic via RT wrote: Show quoted text

>With a German locale (LC_ALL=de_DE.UTF-8) and perl 5.27.2 I see the >following failures:

Curious. I can reproduce that, coming in between 5.27.1 and 5.27.2. My pre-release testing was remiss in not covering a non-English locale. Show quoted text

>Quickly scanning through perldelta I don't see an entry which would >explain the change of behavior.

Nor I. But on a quick look through the git log, I see commit ec268cc8df7c7a90811a099d422eef6a31bf9f8b Author: Karl Williamson <khw@cpan.org> Date: Sat Jul 15 12:36:54 2017 -0600 embed.fnc: Fix declaration of my_strerror() This was improperly made public (but the docs indicate it should not be used by the public). Hash-SharedMem relies on my_strerror() to get the 5.21.1+ locale-dependent behaviour for error strings. With it not available, it falls back to older code that gives the pre-5.21.1 behaviour. -zefram

Sat Aug 12 07:57:09 2017 zefram [...] fysh.org - Correspondence added

Subject:	Re: [rt.cpan.org #122775] Test failures with non-English locale and perl 5.27.2
Date:	Sat, 12 Aug 2017 11:43:59 +0100
To:	Slaven_Rezic via RT <bug-Hash-SharedMem [...] rt.cpan.org>
From:	Zefram <zefram [...] fysh.org>

Slaven_Rezic via RT wrote: Show quoted text

>With a German locale (LC_ALL=de_DE.UTF-8) and perl 5.27.2 I see the >following failures:

Curious. I can reproduce that, coming in between 5.27.1 and 5.27.2. My pre-release testing was remiss in not covering a non-English locale. Show quoted text

>Quickly scanning through perldelta I don't see an entry which would >explain the change of behavior.

Nor I. But on a quick look through the git log, I see commit ec268cc8df7c7a90811a099d422eef6a31bf9f8b Author: Karl Williamson <khw@cpan.org> Date: Sat Jul 15 12:36:54 2017 -0600 embed.fnc: Fix declaration of my_strerror() This was improperly made public (but the docs indicate it should not be used by the public). Hash-SharedMem relies on my_strerror() to get the 5.21.1+ locale-dependent behaviour for error strings. With it not available, it falls back to older code that gives the pre-5.21.1 behaviour. -zefram

Sat Aug 12 08:06:21 2017 andreas.koenig.7os6VVqR [...] franz.ak.mind.de - Correspondence added

CC:	ANDK [...] cpan.org
Subject:	Re: [rt.cpan.org #122775] Test failures with non-English locale and perl 5.27.2
Date:	Sat, 12 Aug 2017 12:33:22 +0200
To:	"Slaven_Rezic via RT" <bug-Hash-SharedMem [...] rt.cpan.org>
From:	Andreas Koenig <andreas.koenig.7os6VVqR [...] franz.ak.mind.de>

Show quoted text

>>>>> On Sat, 12 Aug 2017 02:41:56 -0400, "Slaven_Rezic via RT" <bug-Hash-SharedMem@rt.cpan.org> said:

Show quoted text

> @Andreas: a bisect candidate?

Bisect leads to: : ec268cc8df7c7a90811a099d422eef6a31bf9f8b is the first bad commit : commit ec268cc8df7c7a90811a099d422eef6a31bf9f8b : Author: Karl Williamson <khw@cpan.org> : Date: Sat Jul 15 12:36:54 2017 -0600 : : embed.fnc: Fix declaration of my_strerror() BTW, while at it, I discovered that it always fails with the russian locale ru_RU.utf8 also for 5.27.1, 5.26.0, and 5.24.2 (did not test more). -- andreas

Sat Aug 12 13:45:07 2017 zefram [...] fysh.org - Correspondence added

Subject:	Re: [rt.cpan.org #122775] Test failures with non-English locale and perl 5.27.2
Date:	Sat, 12 Aug 2017 18:44:33 +0100
To:	"(Andreas J. Koenig) via RT" <bug-Hash-SharedMem [...] rt.cpan.org>
From:	Zefram <zefram [...] fysh.org>

(Andreas J. Koenig) via RT wrote: Show quoted text

>BTW, while at it, I discovered that it always fails with the russian >locale ru_RU.utf8 also for 5.27.1, 5.26.0, and 5.24.2 (did not test >more).

It fails from 5.21.1 onwards, i.e., from the change in $! behaviour and introduction of my_strerror(). Prior to 5.21.1, $! shows mojibake (regardless of locale pragma), and H:SM's error message, coming from Strerror(), also shows mojibake. They match, so the tests pass. From 5.21.1 onwards, within the locale pragma, $! shows Cyrillic, but H:SM's error message, now coming from my_strerror() which wraps Strerror(), still shows mojibake. Tests fail. (Without the locale pragma, $! shows English, and so does H:SM's error message, due to my_strerror() being sensitive to the pragma. Except on 5.27.2 where H:SM doesn't get to use my_strerror().) Amusingly, the correct Cyrillic in $! only happens for the UTF-8 locale. In ru_RU.iso88595, $! shows a different flavour of mojibake, and H:SM shows the same thing. Looking at the core source, this comes from S_fixup_errno_string() in mg.c. The PV buffer of $! is always set to the bytes returned by my_strerror(), then this function optionally sets the SvUTF8 flag, based not on whether the locale encoding is UTF-8, but on whether the string actually seen *looks like* UTF-8 encoding. How wonderfully half-assed. My personal testing never showed this up because I only tested with German locales, and, by chance, only with errors whose German messages are spelled entirely in ASCII. I was effectively only testing the language issue, not the encoding issue. -zefram