Skip Menu |

This queue is for tickets about the Scalar-List-Utils CPAN distribution.

Report information
The Basics
Id: 96343
Status: open
Priority: 0/
Queue: Scalar-List-Utils

People
Owner: Nobody in particular
Requestors: meyn_ [...] hotmail.com
Cc: DBOOK [...] cpan.org
REHSACK [...] cpan.org
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: first return value error.
Date: Mon, 9 Jun 2014 14:32:50 +0200
To: bug-Scalar-List-Utils [...] rt.cpan.org
From: joffrey romero <meyn_ [...] hotmail.com>
Hello There, I get an issue with List::Util. I wonder if *first *affects the return value : #!/usr/bin/perl -w use strict; use warnings; use utf8; use feature 'say'; use List::Util qw(first); my $hash = {'Jim' => 1, 'HellO WorlD' => 1, 'Baudelaire' => 1}; { my $key = first { 'hello world' eq lc($_) } keys %$hash; say 'key A: '.$key; # prints "Item: hello world" } { my $key = first { 'hello world' eq lc(join('',$_)) } keys %$hash; say 'key B: '.$key; # prints "Item: HellO WorlD" } exit; Bye.
On Mon Jun 09 08:33:02 2014, meyn_@hotmail.com wrote: Show quoted text
> Hello There, > > I get an issue with List::Util.
I think this appears to be a 5.20-specific bug: leo@cel:~/src/perl/Scalar-List-Utils [git] $ perl5.20.0 rt96343.pl key A: hello world key B: HellO WorlD leo@cel:~/src/perl/Scalar-List-Utils [git] $ perl5.18.0 rt96343.pl key A: HellO WorlD key B: HellO WorlD leo@cel:~/src/perl/Scalar-List-Utils [git] $ perl5.10.0 rt96343.pl key A: HellO WorlD key B: HellO WorlD -- Paul Evans
Test case: use Test::More; use List::Util 'first'; my %things = (FooBar => 1); my $first_thing = first { lc $_ eq 'foobar' } keys %things; is $first_thing, 'FooBar', 'found the right key'; done_testing; from #p5p: 14:39:43 <Bram> An easy work around would be to use: my @foo = keys %things; ... = first ... @foo; :/ 14:40:51 <Grinnz_> huh. that does pass the test 14:41:04 <Grinnz_> so why does @foo work and keys %things not 14:41:41 <Bram> my $first_thing = sort { lc $_ eq "foobar"; } keys %things; also appears to work :/ 14:43:14 <hobbs> Grinnz_: keys %things is a list of temps, once they're in an array they're "real" 14:43:23 <Bram> keys %things returns a variable marked as TEMP.. lc on a variable marked as TEMP will do an inplace edit.. If it's in an array it is *not* marked as TEMP so no in place edit is done 14:43:36 <Bram> or what hobbs said.. should've typed faster :) 14:43:55 <Grinnz_> so i guess the next question is, why is it still marked as TEMP when List::Util::first puts it in $_ 14:44:26 <hobbs> I think, because multicall aliases it 14:44:46 <hobbs> or, not multicall but really first when it's using multicall 14:44:58 <hobbs> GvSV(PL_defgv) = args[index]; 14:45:39 <hobbs> that's pointing $_ to the same SV that came from keys, which is still TEMP 14:47:13 <hobbs> so lc thinks it can modify it, and then first does ST(0) = ST(index) to turn the found item into the return value 14:47:27 <hobbs> but at that point it's been modified 14:47:49 <hobbs> lc thinks that it's the end of the line, if it's receiving a temp than no one else can be using it 14:48:00 <hobbs> but it's wrong here 14:49:00 <Bram> Would it be possible for first to remove the TEMP flag? or will that cause other issues? 14:49:33 <hobbs> I'm not certain, my XS isn't that good, but it seems possible 14:49:45 <hobbs> let me try it 14:49:47 <Bram> Or might be easier: increase the refcount 14:50:14 <Bram> all the code does appear to have '&& SvREFCNT(source) == 1'.. so if increase the ref count (and then decreases it again) it might be fine too 14:50:25 <Bram> no idea what the cleanest/best solution is.. 14:51:44 <Grinnz_> for/grep/etc alias the value and don't have the problem, so whatever those are doing? 14:52:52 <hobbs> Bram: yeah, no idea if that's a correct fix, but putting SvREFCNT_inc/dec around the call makes Grinnz_'s test pass 14:54:12 <hobbs> https://gist.github.com/arodland/be002c0abb6e4048eec9 14:54:13 <dipsy> [ listutil.diff · GitHub ] 14:54:51 <hobbs> again, not claiming that I know what I'm doing, consider it an additional test point rather than a fix :) 14:55:04 <Bram> pp_hot.c:3250 appears to contain: "if (SvPADTMP(src)) { src = PL_stack_base[TOPMARK] = sv_mortalcopy(src);" -> so I'm guessing grep makes a mortal copy if it sees a TEMP? 14:55:11 <Bram> (assuming that is the correct code) 14:55:36 <hobbs> yeah, pp_grepwhile should be the right place 14:57:21 <hobbs> PADTMP isn't the same thing though 14:58:15 <Bram> hmm, true 14:59:57 <Bram> grep { Dump($_}; ... } keys %hash; shows: 'REFCNT = 2, FLAGS = (POK,IsCOW,pPOK)"; first { Dump($_); ... }shows 'REFCNT = 1, FLAGS = (TEMP,POK,IsCOW,pPOK)' :/ 15:00:24 <Bram> oh 15:00:43 <Bram> and the code in pp_grepwhile also has: 'SvTEMP_off(src);' on line 3254 15:00:48 <hobbs> looks like grep unconditionally does SvTEMP_off 15:01:29 <hobbs> which, actually, considering what I was hearing last night about the meaning of that flag, should be a good thing to do 15:02:03 <Bram> so List::Util needs to do it too.. and possible other code on CPAN I guess :/ 15:02:18 <hobbs> it doesn't actually make the SV less doomed, it only disables optimizations that assume they can do somethingto it *because* it's doomed 15:04:15 <hobbs> https://gist.github.com/arodland/833693a99c22db1ec868 also causes a test pass 15:04:15 <dipsy> [ listutil2.diff · GitHub ]
here's a test case demonstrating that this can occur in other functions even if they don't return the values. use Test::More; use List::Util 'any'; my %things = (FooBar => 1); my $first_thing; any { lc $_ eq 'foobar'; $first_thing = $_ } keys %things; is $first_thing, 'FooBar', 'found the right key'; done_testing;
I believe this should now be fixed; at least for first and any/all/none/notall. I imagine other functions will also need adding individually too. But if people can at least test this out, I'll see if it's the right fix. -- Paul Evans
Subject: rt96343.patch
diff --git a/ListUtil.xs b/ListUtil.xs index 183f839..0b15c69 100644 --- a/ListUtil.xs +++ b/ListUtil.xs @@ -442,7 +442,10 @@ CODE: PUSH_MULTICALL(cv); for(index = 1 ; index < items ; index++) { - GvSV(PL_defgv) = args[index]; + SV *def_sv = GvSV(PL_defgv) = args[index]; +# ifdef SvTEMP_off + SvTEMP_off(def_sv); +# endif MULTICALL; if(SvTRUEx(*PL_stack_sp)) { # ifdef PERL_HAS_BAD_MULTICALL_REFCOUNT @@ -510,7 +513,10 @@ PPCODE: PERL_UNUSED_VAR(newsp); PUSH_MULTICALL(cv); for(index = 1; index < items; index++) { - GvSV(PL_defgv) = args[index]; + SV *def_sv = GvSV(PL_defgv) = args[index]; +# ifdef SvTEMP_off + SvTEMP_off(def_sv); +# endif MULTICALL; if(SvTRUEx(*PL_stack_sp) ^ invert) { diff --git a/t/rt-96343.t b/t/rt-96343.t new file mode 100644 index 0000000..5328a41 --- /dev/null +++ b/t/rt-96343.t @@ -0,0 +1,35 @@ +#!/usr/bin/perl + +use strict; +use warnings; + +use Test::More; + +{ + use List::Util qw( first ); + + my $hash = { + 'HellO WorlD' => 1, + }; + + is( ( first { 'hello world' eq lc($_) } keys %$hash ), "HellO WorlD", + 'first (lc$_) perserves value' ); +} + +{ + use List::Util qw( any ); + + my $hash = { + 'HellO WorlD' => 1, + }; + + my $var; + + no warnings 'void'; + any { lc($_); $var = $_; } keys %$hash; + + is( $var, 'HellO WorlD', + 'any (lc$_) leaves value undisturbed' ); +} + +done_testing;
The new rt-96343 test uses syntax from Test::More that changes the dependency version.
On Tue Feb 02 19:46:26 2016, PEVANS wrote: Show quoted text
> I believe this should now be fixed; at least for first and > any/all/none/notall. > > I imagine other functions will also need adding individually too. But > if people can at least test this out, I'll see if it's the right fix.
Sorry for noticing this so late. SvTEMP_off is really not the right fix here. Typeglob slots hold a reference count, so a simple GvSV(PL_defgv) = ... without incrementing the reference count will throw refcount bookkeeping off: Scalar-List-Utils-1.45-EG9lw0$ perl5.24.0 -lMblib -MList::Util=first -e 'my %things = (FooBar => 1); my $first_thing = first { my $ret = lc $_ eq 'foobar'; undef *_; $ret } keys %things; print $first_thing' panic: attempt to copy freed scalar 7f94130056f0 to 7f941302f8a8 at -e line 1. Attempt to free unreferenced scalar: SV 0x7f94130056f0. So when the thingy is assigned to GvSV(...), it really does needs its reference count increased. lc is correct in assuming that SvTEMP && SvREFCNT==1 means the sv is not in use elsewhere. If $_ has a reference count of 1, then it is correct to assume that *_ has the only reference to it, but in this case the mortals stack also holds a reference, so the reference count is wrong.
I played a lot around - I didn't want to release L::MU knowing using a wrong solution. I figured out that if(!GvSV(PL_defgv)) { gv_SVadd(PL_defgv); } sv_setsv_mg(GvSV(PL_defgv), args[i]); MULTICALL; I don't know whether this one is better than SvTEMP_off() - so I'm open to get opinions.