Skip Menu |

This queue is for tickets about the JSON-XS CPAN distribution.

Report information
The Basics
Id: 93307
Status: resolved
Priority: 0/
Queue: JSON-XS

People
Owner: Nobody in particular
Requestors: calle [...] lysator.liu.se
Cc: ether [...] cpan.org
AdminCc:

Bug Information
Severity: Important
Broken in: (no value)
Fixed in: (no value)



Subject: Locale problem with JSON::XS in bleadperl
Commit bc8ec7cc020d0562094a551b280fd3f32bf5eb04 to Perl (included in versions 5.19.8 and 5.19.9) causes JSON::XS to emit JSON that is affected by the setting of the LC_NUMERIC environment variable. With the variable set to, for example, sv_SE.UTF-8 on OSX the produced JSON has commas instead of points in floating-point numbers. This, obviously, results in invalid JSON that cannot later be decoded. I'm not sure if this is a bug in Perl, or a change that JSON::XS needs to adapt to.
On Mon Feb 24 05:38:44 2014, CDYBED wrote: Show quoted text
> Commit bc8ec7cc020d0562094a551b280fd3f32bf5eb04 to Perl (included in > versions 5.19.8 and 5.19.9) causes JSON::XS to emit JSON that is > affected by the setting of the LC_NUMERIC environment variable. With > the variable set to, for example, sv_SE.UTF-8 on OSX the produced JSON > has commas instead of points in floating-point numbers. This, > obviously, results in invalid JSON that cannot later be decoded. > > I'm not sure if this is a bug in Perl, or a change that JSON::XS needs > to adapt to.
Just to cross-reference: The Perl ticket you submitted is https://rt.perl.org/Ticket/Display.html?id=121317 And to note the problem still exists in 5.19.10. Thing is, discussion there (which is still going on) seems to be whether JSON::XS is using an undocumented interface (which divides into two sub-groups, "... and they shouldn't do that" and "... but we shouldn't break existing code"), or whether JSON::XS was broken all along and the change to Perl only exposed the problem. Wish I had enough XS-fu to come up with a patch to JSON::XS. I'd feel a lot better about the whole situation.
Show quoted text
> Thing is, discussion there (which is still going on) seems to be > whether JSON::XS is using an undocumented interface (which divides > into two sub-groups, "... and they shouldn't do that" and "... but we > shouldn't break existing code"), or whether JSON::XS was broken all > along and the change to Perl only exposed the problem.
Here is a summary of the situation, as I've understood it from following p5p: * JSON::XS is one of a number of CPAN modules that use the macro Gconvert. * Gconvert is semi-public, in that it is not in perlapi but it is in some other API documentation. * Gconvert is implemented by calling system functions that on most platforms obey locale rules. This means that it has always had the potential to produce strings with decimal-comma instead of decimal-point. * What happened in the commit mentioned in my bug report was that Perl started calling setlocale() predictably at startup. That means that when run in locales with decimal-comma Gconvert will emit such strings all the time rather than very rarely. * This is listed as a blocking bug for the Perl 5.20 release. * The suggested solution is to split Gconvert into two macros, one that follows locale rules and one that ignores them. These macros will be documented in perlapi as parts of the public XS API. Backwards compatibility will be provided via ppport.h * For JSON::XS this means that after Perl 5.20 is released, a new version of JSON::XS should be released the uses the new, documented interface instead of Gconvert.
Subject: Re: [rt.cpan.org #93307] Locale problem with JSON::XS in bleadperl
Date: Thu, 27 Mar 2014 16:04:11 +0100
To: Tom Wyant via RT <bug-JSON-XS [...] rt.cpan.org>
From: Marc Lehmann <schmorp [...] schmorp.de>
On Wed, Mar 26, 2014 at 12:46:28PM -0400, Tom Wyant via RT <bug-JSON-XS@rt.cpan.org> wrote: Show quoted text
> And to note the problem still exists in 5.19.10. > > Thing is, discussion there (which is still going on) seems to be whether JSON::XS is using an undocumented interface (which divides into two sub-groups, "... and they shouldn't do that" and "... but we shouldn't break existing code"), or whether JSON::XS was broken all along and the change to Perl only exposed the problem.
It's part of the public external interface. If you could only use the documented interface then most XS modules would be impossible to write, as perl's documentation was and always has been incomplete. Show quoted text
> Wish I had enough XS-fu to come up with a patch to JSON::XS. I'd feel a lot better about the whole situation.
I wish it were possible to patch JSON::XS. The talk about undocumented interfaces is bullshit, of course - the real issue is perl breaking backwards compatibility, as guaranteed by perlpolicy and common sense. p5p *need* to stop breaking large parts of CPAN in every relase, or at least update perlpolicy to honestly say that backwards compatibility is a non-goal, in which I will happily stop supporting newer perl versions in my modules, something that I am close to do, because I simply cannot keep up with the breakage introduced by p5p and the dishonest claims of backwards compatibility when every regression I report is effectively ignored. -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schmorp@schmorp.de -=====/_/_//_/\_,_/ /_/\_\
Here is a short script that demonstrates the problem (you may have to edit it to use a suitable locale for your machine). A quick test with the Perl versions I have installed at the moment shows the problem appearing somewhere between 5.14.4 and 5.16.3.
Subject: bugtest.pl
#!/usr/bin/env perl use strict; use warnings; use POSIX qw[setlocale LC_NUMERIC]; BEGIN { setlocale( LC_NUMERIC, 'sv_SE.UTF-8' ) } use JSON::XS; my $json = encode_json({ foo => 1.23}); if ($json eq '{"foo":1.23}') { print "OK\n"; exit 1; } elsif ($json eq '{"foo":1,23}') { print "NOT OK\n"; exit 0; } else { print "Weird: $json\n"; }
Subject: Re: [rt.cpan.org #93307] Locale problem with JSON::XS in bleadperl
Date: Fri, 28 Mar 2014 16:14:12 +0100
To: Calle Dybedahl via RT <bug-JSON-XS [...] rt.cpan.org>
From: Marc Lehmann <schmorp [...] schmorp.de>
On Fri, Mar 28, 2014 at 03:41:16AM -0400, Calle Dybedahl via RT <bug-JSON-XS@rt.cpan.org> wrote: Show quoted text
> Here is a short script that demonstrates the problem (you may have to edit it to use a suitable locale for your machine). A quick test with the Perl versions I have installed at the moment shows the problem appearing somewhere between 5.14.4 and 5.16.3.
That's not the real problem. At least up to 5.18, calling POSIX::setlocale enabling LC_NUMERIC causes a lot of issues with perl itself (and the effect on JSON::XS is documented). For example, in 5.18, this would not work as expected by a lot of code, printing 1 instead of 1.2: my $x = 0.2; print "$x"+1; So locale breaking perl code is not a new thing. Superficially, the above is a bug in perl, as there is no such thing as a string type or a number type, so the stringified form must work correctly as a number. And it's quite obvious, some vague statements to the contrary, that perl numbers must stringify with ".", as witnessed by perlfaq4 for example, which also breaks down: when( /^-?(?:\d+\.?|\.\d)\d*\z/ ) { say "\tis a real number"; continue } when( /^[+-]?(?=\.?\d)\d*\.?\d*(?:e[+-]?\d+)?\z/i) { say "\tis a C float" } The assumption that numbers stringify in a way compatible to source code is deeply ingrained, and also very useful. Of course, it's understandable that perl worked like this for, like 20+ years: changing the locale around stringifying numbers is not thread safe, and very slow (I hope perl doesn't try to do that in 5.20, thraeds are too useful to be ruled out by perl), and converting floating point numbers yourself is not possible in portable code, so you are stuck with the libc, which heeds the locale. So while the real problem is the unusability if locales to solve real problems, this problem cannot be solved. That leaves us with perl creating such problems on it's own, and that is perl enabling the LC_NUMERIC category with "use locale" now apparently. Or something else to the extent. I mean, perl itself didn't work correctly for all of its life, so why not enable it, because surely, everybody elses code is magically going to cope with the very same problem that perl didn't for decades. In fact, I do not care much about what causes the problem, I do care that basically all my bugreports about such regressions are met with either uncivil and/or bullshit answers, or outright denial that the problem even exists. While I will happily explain problems and report bugs, I have given up on trying to reason with p5pers, as they have clearly given up on backwards compatibility (and in fact, some of the former p5pers have done so, too). So, the *real* problem is not that this happens - bugs happen. The *real* problem is that the bug isn't immediately fixed, but instead it's always such a fight to get regressions acknowledged, so they often end up in releases, which is then used to argue that backwards compatibility is not relevant anymore (which gave us, among other things, the unicode mess in perl). -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / schmorp@schmorp.de -=====/_/_//_/\_,_/ /_/\_\