Bug #133128 for List-MoreUtils-XS: Tests in t/xs/minmaxstr.t are failing on Ubuntu 20.04, perl-5.28.3, LANG=nb

Fri Aug 07 10:27:50 2020 sjn [...] cpan.org - Ticket created

Subject:

Tests failing on Ubuntu 20.04, perl-5.28.3

Something weird is going on here.

------------------------------------------
$ make test
"/home/sjn/perl5/perlbrew/perls/perl-5.28.3/bin/perl" -MExtUtils::Command::MM -e 'cp_nonempty' -- XS.bs blib/arch/auto/List/MoreUtils/XS/XS.bs 644
PERL_DL_NONLAZY=1 "/home/sjn/perl5/perlbrew/perls/perl-5.28.3/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/xs/*.t xt/*.t
t/xs/after.t ................ ok
t/xs/after_incl.t ........... ok
t/xs/all.t .................. ok
t/xs/all_u.t ................ ok
t/xs/any.t .................. ok
t/xs/any_u.t ................ ok
t/xs/apply.t ................ ok
t/xs/arrayify.t ............. ok
t/xs/before.t ............... ok
t/xs/before_incl.t .......... ok
t/xs/binsert.t .............. ok
t/xs/bremove.t .............. ok
t/xs/bsearch.t .............. ok
t/xs/bsearchidx.t ........... ok
t/xs/duplicates.t ........... ok
t/xs/each_array.t ........... ok
t/xs/equal_range.t .......... ok
t/xs/false.t ................ ok
t/xs/firstidx.t ............. ok
t/xs/firstres.t ............. ok
t/xs/firstval.t ............. ok
t/xs/frequency.t ............ ok
t/xs/Import.t ............... ok
t/xs/indexes.t .............. ok
t/xs/insert_after.t ......... ok
t/xs/insert_after_string.t .. ok
t/xs/lastidx.t .............. ok
t/xs/lastres.t .............. ok
t/xs/lastval.t .............. ok
t/xs/listcmp.t .............. ok
t/xs/lower_bound.t .......... ok
t/xs/mesh.t ................. ok
t/xs/minmax.t ............... ok
t/xs/minmaxstr.t ............ 1/?
Show quoted text

#   Failed test at t/xs/minmaxstr.t line 15.
#          got: 'AB'
#     expected: 'AA'

Show quoted text

#   Failed test at t/xs/minmaxstr.t line 16.
#          got: 'AA'
#     expected: 'ZZ'

Show quoted text

#   Failed test at t/xs/minmaxstr.t line 21.
#          got: 'AB'
#     expected: 'AA'

Show quoted text

#   Failed test at t/xs/minmaxstr.t line 22.
#          got: 'AA'
#     expected: 'ZZ Top'
# Looks like you failed 4 tests of 11.

t/xs/minmaxstr.t ............ Dubious, test returned 4 (wstat 1024, 0x400)
Failed 4/11 subtests
t/xs/mode.t ................. ok
t/xs/natatime.t ............. ok
t/xs/none.t ................. ok
t/xs/none_u.t ............... ok
t/xs/notall.t ............... ok
t/xs/notall_u.t ............. ok
t/xs/occurrences.t .......... ok
t/xs/one.t .................. ok
t/xs/one_u.t ................ ok
t/xs/onlyidx.t .............. ok
t/xs/onlyres.t .............. ok
t/xs/onlyval.t .............. ok
t/xs/pairwise.t ............. ok
t/xs/part.t ................. ok
t/xs/qsort.t ................ ok
t/xs/reduce_0.t ............. ok
t/xs/reduce_1.t ............. ok
t/xs/reduce_u.t ............. ok
t/xs/samples.t .............. ok
t/xs/singleton.t ............ ok
t/xs/true.t ................. ok
t/xs/uniq.t ................. ok
t/xs/upper_bound.t .......... ok
t/xs/XS.t ................... skipped: Unreasonable unless loaded via List::MoreUtils
t/xs/zip6.t ................. ok

Test Summary Report
-------------------
t/xs/minmaxstr.t          (Wstat: 1024 Tests: 11 Failed: 4)
Failed tests: 1-4
Non-zero exit status: 4
Files=59, Tests=4507, 5 wallclock secs ( 0.22 usr 0.05 sys + 4.33 cusr 0.32 csys = 4.92 CPU)
Result: FAIL
Failed 1/59 test programs. 4/4507 subtests failed.
make: *** [Makefile:1092: test_dynamic] Error 255

------------------------------------------
$ perl -v

This is perl 5, version 28, subversion 3 (v5.28.3) built for x86_64-linux
(with 1 registered patch, see perl -V for more detail)

------------------------------------------
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.3.0-10ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)

------------------------------------------
$ uname -a
Linux smallworld 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

------------------------------------------

Please tell me if there is something I can do to track down the reason behind this. :-)

- Salve

Fri Aug 07 10:34:13 2020 sjn [...] cpan.org - Subject changed from 'Tests failing on Ubuntu 20.04, perl-5.28.3' to 'Tests in t/xs/minmaxstr.t are failing on Ubuntu 20.04, perl-5.28.3'

Fri Aug 07 11:33:08 2020 sjn [...] cpan.org - Correspondence added

Found something interesting. Apparently, when LANG is set to something else than en_US.utf8, the test faiils

Fri Aug 07 11:34:12 2020 sjn [...] cpan.org - Correspondence added

fr. 07. aug. 2020 17.33.08 skrev sjn:
Show quoted text

> Found something interesting. Apparently, when LANG is set to something else
> than en_US.utf8, the test faiils

LANG=C also works.

Fri Aug 07 11:35:14 2020 REHSACK [...] cpan.org - Correspondence added

Hi Salve, I have no idea what you did or how to reproduce. A little bit more details would be nice. What is that offload-target you are using? Is there some internal out-of-order processing of loops and maybe a return is wrong reported? I have no idea about details of what you did. When I run into such issues, I do printf() debugging (read: write logs). Cheers, Jens

Fri Aug 07 11:35:15 2020 The RT System itself - Status changed from 'new' to 'open'

Fri Aug 07 11:37:12 2020 REHSACK [...] cpan.org - Correspondence added

Since I do not do any gettext() or nls processing, the issue is likely outside that code.

Fri Aug 07 11:41:43 2020 sjn [...] cpan.org - Correspondence added

fr. 07. aug. 2020 17.34.12 skrev sjn:
Show quoted text

> fr. 07. aug. 2020 17.33.08 skrev sjn:

> > Found something interesting. Apparently, when LANG is set to
> > something else
> > than en_US.utf8, the test faiils

>
> LANG=C also works.

If the test forces LC_COLLATE=C then it succeeds even if LANG is set to something weird- :)

Fri Aug 07 11:43:45 2020 rehsack [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #133128] Tests in t/xs/minmaxstr.t are failing on Ubuntu 20.04, perl-5.28.3
Date:	Fri, 7 Aug 2020 17:43:37 +0200
To:	"Salve J. Nilsen via RT" <bug-List-MoreUtils-XS [...] rt.cpan.org>
From:	Jens Rehsack <rehsack [...] gmail.com>

Show quoted text

> Am 07.08.2020 um 17:41 schrieb Salve J. Nilsen via RT <bug-List-MoreUtils-XS@rt.cpan.org>: > > Queue: List-MoreUtils-XS > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=133128 > > > fr. 07. aug. 2020 17.34.12 skrev sjn:

>> fr. 07. aug. 2020 17.33.08 skrev sjn:

>>> Found something interesting. Apparently, when LANG is set to >>> something else >>> than en_US.utf8, the test faiils

>> >> LANG=C also works.

> > > If the test forces LC_COLLATE=C then it succeeds even if LANG is set to > something weird- :)

That sounds like: If you do something XXX to break other weird behavior, you're not affected. I can plan doing that - but it's very low priority. Cheers -- Jens Rehsack - rehsack@gmail.com

Download signature.asc
application/pgp-signature 833b

Message body not shown because it is not plain text.

Fri Aug 07 14:50:25 2020 sjn [...] cpan.org - Correspondence added

fr. 07. aug. 2020 17.43.45 skrev rehsack@gmail.com:
> >
> > If the test forces LC_COLLATE=C then it succeeds even if LANG is set
> > to something weird- :)
>
> That sounds like:
> If you do something XXX to break other weird behavior, you're not
> affected.
>
> I can plan doing that - but it's very low priority.

I think I know what's going on now: https://en.wikipedia.org/wiki/%C3%85#Danish_and_Norwegian

When following Norwegian sorting rules, there is an exception for the letters "aa" - they are sorted with the letter å, which is at the *end* of the alphabet, and not the beginning. "Å" and "Aa" are considered the same letter/sound for historical reasons.

There's a nice overview of this issue here: https://en.wikipedia.org/wiki/%C3%85#Danish_and_Norwegian

The Norwegian alphabet goes like this: A, B, C, ..., X, Y, Z, Æ, Ø, Å.

And since Å is the same as AA, and you sort with Norwegian collation rules, AA .. ZZ becomes AB .. ZZ, AA - which is what the failing tests show.

In other words, the tests are sensitive to differences in collation rules.

One fix would be to *not* use AA in the data you want to sort - which would solve the problem on systems using Norwegian or Danish collation rules, but not on all systems.

The better fix would be to specify which sorting rules you are actually using in your test.

Sat Aug 08 08:51:19 2020 sjn [...] cpan.org - Subject changed from 'Tests in t/xs/minmaxstr.t are failing on Ubuntu 20.04, perl-5.28.3' to 'Tests in t/xs/minmaxstr.t are failing on Ubuntu 20.04, perl-5.28.3, LANG=nb_NO.utf8'

Sat Aug 08 09:04:03 2020 sjn [...] cpan.org - Correspondence added

fr. 07. aug. 2020 20.50.25 skrev sjn:
Show quoted text

>
> I think I know what's going on now:
> https://en.wikipedia.org/wiki/%C3%85#Danish_and_Norwegian

To reproduce:

Show quoted text

# --> Pick one of the locales or collation rules below.
# All make the test fail while sorting correctly for that locale. Note
# the tests only fail when following Danish or Norwegian collation rules.
# Other languages may work fine. YMMV. Sorting/collation rules can be weird!

LANG=da_DK.utf8
LANG=nn_NO.utf8
LC_COLLATE=da_DK.utf8
LC_COLLATE=nb_NO.utf8

Show quoted text

# Fails

make test

Show quoted text

# If you make sure to explicitly require a specific collation rule that matches your expectaions in your test, then it works

LANG=C # sets all locale settings to C
LC_COLLATE=C # sets just the collation rules to C

Show quoted text

# Now it should work

make test

Mon Aug 10 07:22:01 2020 sjn [...] cpan.org - Correspondence added

RT-Send-CC:

rehsack [...] gmail.com

This issue can be fixed by adding the following lines to the top of t/xs/minmaxstr.t (or whatever file it is generated from):

Show quoted text

======================8<-------------------

Show quoted text

# Force collation/sorting to go as define by the C locale. Without it, the test fails
# under Danish and Norwegian locales. Collation can still be overridden by setting the
# LC_ALL environment variable to your locale of choice.

use POSIX qw(setlocale LC_COLLATE);
setlocale(LC_COLLATE, "C");

----------------->8=======================

Hope this helps! :-)

- Salve

Mon Aug 10 07:31:20 2020 REHSACK [...] cpan.org - Correspondence added

I'm kind of unsure. The intention is not to force a specific order. Maybe another (probably better) solution is to use a sorted list instead of forcing a dedicated locale. In such a case, it will guarantee that the Perl behavior which is configured makes the test succeed. What do you think?

Mon Aug 10 12:06:12 2020 sjn [...] cpan.org - Correspondence added

ma. 10. aug. 2020 13.31.20 skrev REHSACK:
> I'm kind of unsure. The intention is not to force a specific order.
>
> Maybe another (probably better) solution is to use a sorted list
> instead of forcing a dedicated locale. In such a case, it will
> guarantee that the Perl behavior which is configured makes the test
> succeed.
>
> What do you think?

Hm.

First of all, what are you actually testing?

I would say that the locale offers a precondition that influences how the tested functions work. If you want a test that actually confirms that a function's behavior/output is as expected, given a defined input and context, then you should definitely set the locale. The locale *does* influence the output, so if you don't care about it for some reason, I'd love to know why - including what the actual purpose of the test is.

- Salve

Mon Aug 10 14:22:48 2020 REHSACK [...] cpan.org - Correspondence added

The test finds the "lowest" and the "highest" string occurrences in a list. When comparing to (sort list)[0] and (sort list)[-1] - it's either fine than setting locale but proves for other locales, too :) Please tell me when I'm mistaken.

Mon Aug 10 14:38:11 2020 sjn [...] cpan.org - Correspondence added

ma. 10. aug. 2020 20.22.48 skrev REHSACK:
Show quoted text

> The test finds the "lowest" and the "highest" string occurrences in a
> list. When comparing to (sort list)[0] and (sort list)[-1] - it's
> either fine than setting locale but proves for other locales, too :)

for other locales too, except for the Norwegian and Danish ones.

Show quoted text

> Please tell me when I'm mistaken.

You are correct IF you assume that people don't run the tests under different locales. I've shown you when this assumption breaks. The solution is to *not* assume a locale, and instead specify a locale explicitly up-front.

If you need to test the behavior of the functions under different locales, then do so by explicitly stating which locales you want to use before each of your tests.

Mon Aug 10 15:37:29 2020 REHSACK [...] cpan.org - Correspondence added

I'll have to look at the code :D There're two solutions and one (yours) is proven.

Fri Oct 02 05:01:47 2020 REHSACK [...] cpan.org - Correspondence added

Suggested patch applied, fixed with next release.

Fri Oct 02 05:01:48 2020 REHSACK [...] cpan.org - Status changed from 'open' to 'resolved'

Fri Oct 02 05:48:29 2020 sjn [...] cpan.org - Correspondence added

fr. 02. okt. 2020 11.01.47 skrev REHSACK:
Show quoted text

> Suggested patch applied, fixed with next release.

Thanks! Are you planning to do one soon?

Fri Oct 02 06:32:41 2020 REHSACK [...] cpan.org - Correspondence added

dev release is out, LMU (PP) comes next (dev), then both regular in a week if nothing happens

Bug #133128 for List-MoreUtils-XS: Tests in t/xs/minmaxstr.t are failing on Ubuntu 20.04, perl-5.28.3, LANG=nb_NO.utf8