Skip Menu |

This queue is for tickets about the threads-shared CPAN distribution.

Report information
The Basics
Id: 79906
Status: resolved
Priority: 0/
Queue: threads-shared

People
Owner: Nobody in particular
Requestors: philippe.causse [...] nsn.com
Cc:
AdminCc:

Bug Information
Severity: Wishlist
Broken in: 1.41
Fixed in: 1.42



Subject: numeric part of a dualvar not preserved via shared array / shared hash
Date: Fri, 28 Sep 2012 14:15:43 +0200
To: bug-threads-shared [...] rt.cpan.org
From: Philippe Causse <philippe.causse [...] nsn.com>
Hello, please find the following bug-report: Versions: ====== * perl 5.16.0 * threads::shared 1.4 Problem description: ============= When storing a dualvar in a shared array (or shared hash), only the SVPV part is preserved. The IV part is lost. Storing via a shared scalar works fine though. EXAMPLE CODE: ========== #!/usr/bin/env perl use 5.016; use strict; use warnings; use threads; use threads::shared; use Scalar::Util qw( dualvar ); use constant FOURTY_TWO => dualvar( 42, 'Fourty-two' ); my $ss : shared; my @sa : shared; my %sh : shared; $ss = FOURTY_TWO; $sa[0] = FOURTY_TWO; $sh{key} = FOURTY_TWO; printf "Scalar ok: %s\n", $ss == FOURTY_TWO ? 'YES' : 'NO'; printf "Array ok: %s\n", $sa[0] == FOURTY_TWO ? 'YES' : 'NO'; printf "Hash ok: %s\n", $sh{key} == FOURTY_TWO ? 'YES' : 'NO'; #---------------------------------------------------------------------------------------------------- Execution shows: $ ./bugtest.pl Scalar ok: YES Argument "Fourty-two" isn't numeric in numeric eq (==) at ./bugtest.pl line 22. Array ok: NO Argument "Fourty-two" isn't numeric in numeric eq (==) at ./bugtest.pl line 23. Hash ok: NO
On 2012-09-28 08:15:57, philippe.causse@nsn.com wrote: Show quoted text
> When storing a dualvar in a shared array (or shared hash), > only the SVPV part is preserved. The IV part is lost. > > Storing via a shared scalar works fine though.
Interesting. dualvar() plays tricks with a scalar such that the scalar is no longer a "basic" Perl type, nor is it an object. Thus, threads::shared doesn't copy it "correctly" when adding it to a shared structure. A dualvar() passed on a Thread::Queue doesn't come out dualvar() on the other end either. Setting a shared scalar to a dualvar() works because the dualvar() is not copied; its refcount is just incremented. This is really a bug against Scalar::Util as it's that module that is implementing the dualvar() feature.
Subject: Re: [rt.cpan.org #79906] numeric part of a dualvar not preserved via shared array / shared hash
Date: Fri, 28 Sep 2012 15:41:14 +0200
To: bug-threads-shared [...] rt.cpan.org
From: Philippe Causse <philippe.causse [...] nsn.com>
Hello Jerry, Thank you for taking the time answering this. I agree that dualvar is "playing tricks", although a dualvar it is still a basic type (it's an SV for with IOK / NOK is set and POK is also set). When an Sv is upgraded to an SvPV, the NV slot is still available and that's what dualvar() uses. By curiosity, I have tested against the most famous dualvar of Perl: $! and obtained slightly different results. Example of application: if you have a thread that wants to report an error via message passing (let's say via Thread::Queue), you would need to 'split' your error code and reason into two parts (or chose to only relay the numeric form). use Errno qw( EINTR ); $! = EINTR; my $ss : shared; my @sa : shared; my $ns; $ns = $!; $ss = $!; $sa[0] = $!; printf "Via non-shared scalar: %d/%s\n", $ns, $ns; printf "Via shared scalar: %d/%s\n", $ss, $ss; printf "Via shared array: %d/%s\n", $sa[0], $sa[0]; ------------------------ GIVES: Via non-shared scalar: 4/Interrupted system call Argument "Interrupted system call" isn't numeric in printf at ./bugtest.pl line 29. Via shared scalar: 0/Interrupted system call Argument "Interrupted system call" isn't numeric in printf at ./bugtest.pl line 30. Via shared array: 0/Interrupted system call Then examining the data with Devel::Peek reveals interesting details: ===> the non-shared scalar ($ns) : SV = PVMG(0xa1e6f44) at 0xa2186d8 REFCNT = 1 FLAGS = (PADMY,POK,pNOK,pPOK) IV = 0 NV = 4 PV = 0xa207170 "Interrupted system call"\0 CUR = 23 LEN = 24 ===> The shared scalar ($ss): SV = PVMG(0xa1e6f0c) at 0xa184078 REFCNT = 1 FLAGS = (PADMY,GMG,SMG,pPOK) IV = 0 NV = 4 PV = 0xa2002c8 "Interrupted system call"\0 CUR = 23 LEN = 24 MAGIC = 0xa21bfc0 MG_VIRTUAL = 0xb728f300 MG_TYPE = PERL_MAGIC_shared_scalar(n) MG_FLAGS = 0x30 DUP LOCAL MG_PTR = 0xa1ed8e0 "" You can see that after assignment the NV part of the non-shared scalar still contains 4 in its NV, but the pNOK flag has been cleared. Only the pPOK flag is left. Via non-shared scalar: 4/Interrupted system call Argument "Interrupted system call" isn't numeric in printf at ./bugtest.pl line 31. Via shared scalar: 0/Interrupted system call On 09/28/2012 03:17 PM, ext Jerry D. Hedden via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=79906 > > > On 2012-09-28 08:15:57, philippe.causse@nsn.com wrote:
>> When storing a dualvar in a shared array (or shared hash), >> only the SVPV part is preserved. The IV part is lost. >> >> Storing via a shared scalar works fine though.
> Interesting. dualvar() plays tricks with a scalar such that the scalar > is no longer a "basic" Perl type, nor is it an object. Thus, > threads::shared doesn't copy it "correctly" when adding it to a shared > structure. A dualvar() passed on a Thread::Queue doesn't come out > dualvar() on the other end either. Setting a shared scalar to a > dualvar() works because the dualvar() is not copied; its refcount is > just incremented. > > This is really a bug against Scalar::Util as it's that module that is > implementing the dualvar() feature.
On 2012-09-28 09:41:28, philippe.causse@nsn.com wrote: Show quoted text
> Hello Jerry, > > Thank you for taking the time answering this. > > I agree that dualvar is "playing tricks", although a dualvar it is > still a basic type (it's an SV for with IOK / NOK is set and POK is
also Show quoted text
> set). > > When an Sv is upgraded to an SvPV, the NV slot is still available and > that's what dualvar() uses. > > By curiosity, I have tested against the most famous dualvar of Perl: > $! and obtained slightly different results.
$! is a special scalar that uses 'magic' to make it a dualvar - the same sort of 'trick' is used for it as for what Scalar::Util does. From mg.c for setting $!: SvNOK_on(sv); /* what a wonderful hack! */ I do not disagree that it would be nice for threads::shared to preserve the dualvar nature of scalars so set, but that 'nature' is not generally supported by the Perl system and only works with special coding. I'll leave this ticket open on the wishlist for future reference.
Subject: Re: [rt.cpan.org #79906] numeric part of a dualvar not preserved via shared array / shared hash
Date: Fri, 28 Sep 2012 16:51:58 +0200
To: bug-threads-shared [...] rt.cpan.org
From: Philippe Causse <philippe.causse [...] nsn.com>
Thanks ;-)
CC: jdhedden [...] cpan.org
Subject: Re: [rt.cpan.org #79906] numeric part of a dualvar not preserved via shared array / shared hash
Date: Sun, 30 Sep 2012 20:07:57 +0200
To: bug-threads-shared [...] rt.cpan.org
From: "Causse, Philippe (NSN - DK/Copenhagen)" <philippe.causse [...] nsn.com>
Hello Jerry, I have found the reason why dualvars don't work for shared arrays: During the call to Perl_sharedsv_associate(), the SV public flags IOK or NOK are lost, only the POK gets transferred to the shared SV. This is happens after the call to sv_magicext() where IOK or POK become pIOK and pPOK respectively. When the SvPVLV becomes magical, it loses its "duality". Checking the flags on the shared SV when sharedsv_elem_mg_STORE is called, I noticed that either IOK or POK were set, but never both. So, the workaround is to copy the IOK (or NOK) flag from the proxy SV onto the shared SV, when SvPOK(ssv) is true. The following patch does the job and all tests succeed (i.e. no bad side effects): *** threads-shared-1.41/shared.xs 2012-09-05 19:20:53.000000000 +0200 --- threads-shared-1.41_PHIL/shared.xs 2012-09-30 19:52:46.000000000 +0200 *************** *** 963,970 **** svp = hv_fetch((HV*) saggregate, key, len, 1); } CALLER_CONTEXT; ! Perl_sharedsv_associate(aTHX_ sv, *svp); sharedsv_scalar_store(aTHX_ sv, *svp); LEAVE_LOCK; return (0); } --- 963,977 ---- svp = hv_fetch((HV*) saggregate, key, len, 1); } CALLER_CONTEXT; ! { ! U32 dualvar_flags = SvFLAGS(sv) & ( SVf_IOK | SVf_NOK ); ! Perl_sharedsv_associate(aTHX_ sv, *svp); /* this loses IOK/NOK & POK ! */ sharedsv_scalar_store(aTHX_ sv, *svp); + if( SvPOK( *svp) ) { + SvFLAGS(*svp) |= dualvar_flags; /* restore dualvar flags */ + } + } + LEAVE_LOCK; return (0); } And adding t/sv_dualvar.t to the test suite: use strict; use warnings; BEGIN { use Config; if ( !$Config{'useithreads'} ) { print("1..0 # SKIP Perl not compiled with 'useithreads'\n"); exit(0); } } use ExtUtils::testlib; sub ok { my ( $id, $ok, $name ) = @_; # You have to do it this way or VMS will get confused. if ($ok) { print("ok $id - $name\n"); } else { print("not ok $id - $name\n"); printf( "# Failed test at line %d\n", (caller)[2] ); } return ($ok); } BEGIN { $| = 1; print("1..17\n"); ### Number of tests that will be run ### } use Scalar::Util qw( dualvar ); use Devel::Peek; use threads; use threads::shared; ok( 1, 1, 'Loaded' ); ### Start of Testing ### my $dv = dualvar( 42, 'Fourty-Two' ); my $pi = dualvar( 3.14, 'PI' ); my @a : shared; # Individual assignment # Verify that dualvar preserved during individual element assignment $a[0] = $dv; $a[1] = $pi; ok( 2, $a[0] == 42, 'IV number preserved' ); ok( 3, $a[0] eq 'Fourty-Two', 'string preserved' ); ok( 4, $a[1] == 3.14, 'NV number preserved' ); ok( 5, $a[1] eq 'PI', 'string preserved' ); #-- List initializer # Verify that dualvar preserved during initialization my @a2 : shared = ( $dv, $pi ); ok( 6, $a2[0] == 42, 'IV number preserved' ); ok( 7, $a2[0] eq 'Fourty-Two', 'string preserved' ); ok( 8, $a2[1] == 3.14, 'NV number preserved' ); ok( 9, $a2[1] eq 'PI', 'string preserved' ); #-- List assignment # Verify that dualvar preserved during list assignment my @a3 : shared = ( 0, 0 ); @a3 = ( $dv, $pi ); ok( 10, $a3[0] == 42, 'IV number preserved' ); ok( 11, $a3[0] eq 'Fourty-Two', 'string preserved' ); ok( 12, $a3[1] == 3.14, 'NV number preserved' ); ok( 13, $a3[1] eq 'PI', 'string preserved' ); # Back to non-shared # Verify that entries are still dualvar when leaving the array my @nsa = @a3; ok( 14, $nsa[0] == 42, 'IV number preserved' ); ok( 15, $nsa[0] eq 'Fourty-Two', 'string preserved' ); ok( 16, $nsa[1] == 3.14, 'NV number preserved' ); ok( 17, $nsa[1] eq 'PI', 'string preserved' ); # Transmission of $! won't work because it's a tied SV # so, sharing $! can't be done. However, it can be "ticked" # by creating a dualvar from it, which would remove the magic. # if(0) { #----- INFORMATIONAL, not part of test. $! = 33; # my $ss : shared = $!; ## <<== Won't work my @sa : shared = dualvar($!,$!); # works ! my ($x ,) = @sa; printf "X = %d/%s\n", $x, $x; } exit(0);
On 2012-09-30 14:08:14, philippe.causse@nsn.com wrote: Show quoted text
> The following patch does the job and all tests succeed (i.e. no bad > side effects):
This is most excellent. I have updated the module per your patch, and have sent a patch to blead. When that is accepted, I'll propagate to CPAN. Thanks.
On 2012-10-01 09:19:27, JDHEDDEN wrote: Show quoted text
> On 2012-09-30 14:08:14, philippe.causse@nsn.com wrote:
> > The following patch does the job and all tests succeed (i.e. no bad > > side effects):
> > This is most excellent. I have updated the module per your patch, and > have sent a patch to blead. When that is accepted, I'll propagate to > CPAN. Thanks.
It turns out that there are several other places where this type of logic needs to be applied (e.g., push, pop). I'm not going to release to CPAN until I have it all down.
Subject: Re: [rt.cpan.org #79906] numeric part of a dualvar not preserved via shared array / shared hash
Date: Tue, 02 Oct 2012 00:36:13 +0200
To: bug-threads-shared [...] rt.cpan.org
From: "Causse, Philippe (NSN - DK/Copenhagen)" <philippe.causse [...] nsn.com>
On 01/10/12 23.03, ext Jerry D. Hedden via RT wrote: Show quoted text
> It turns out that there are several other places where this type of > logic needs to be applied (e.g., push, pop). I'm not going to release > to CPAN until I have it all down.
Sorry, I have overlooked those. The good news is that they are easily located and fixed. Only PUSH and UNSHIFT required the extra work. I have also checked operations on hashes and, as expected, it worked fine. So basically I checked all routines calling sharedsv_scalar_store() [which is basically where the problem happens] and added the fix as required. Please find below a new context diff against shared.xs 1.41 (including previously submitted changed), test code follows. I believe that you have it all down now. Best regards, /Philippe. phil$ diff -c ~/threads-shared-1.41/shared.xs shared.xs *** /Users/phil/threads-shared-1.41/shared.xs 2012-09-05 19:20:53.000000000 +0200 --- shared.xs 2012-10-02 00:17:36.000000000 +0200 *************** *** 963,970 **** svp = hv_fetch((HV*) saggregate, key, len, 1); } CALLER_CONTEXT; ! Perl_sharedsv_associate(aTHX_ sv, *svp); ! sharedsv_scalar_store(aTHX_ sv, *svp); LEAVE_LOCK; return (0); } --- 963,976 ---- svp = hv_fetch((HV*) saggregate, key, len, 1); } CALLER_CONTEXT; ! { ! U32 dualvar_flags = SvFLAGS(sv) & ( SVf_IOK | SVf_NOK ); ! Perl_sharedsv_associate(aTHX_ sv, *svp); ! sharedsv_scalar_store(aTHX_ sv, *svp); ! if(SvPOK(*svp)) { ! SvFLAGS(*svp) |= dualvar_flags; /* restore dualvar flags */ ! } ! } LEAVE_LOCK; return (0); } *************** *** 1266,1275 **** --- 1272,1283 ---- for (i = 1; i < items; i++) { SV* tmp = newSVsv(ST(i)); SV *stmp; + U32 dualvar_flags = SvFLAGS(tmp) & ( SVf_IOK | SVf_NOK ); ENTER_LOCK; stmp = S_sharedsv_new_shared(aTHX_ tmp); sharedsv_scalar_store(aTHX_ tmp, stmp); SHARED_CONTEXT; + if( SvPOK( stmp) ) { SvFLAGS(stmp) |= dualvar_flags; } av_push((AV*) sobj, stmp); SvREFCNT_inc_void(stmp); SHARED_RELEASE; *************** *** 1289,1297 **** --- 1297,1307 ---- CALLER_CONTEXT; for (i = 1; i < items; i++) { SV *tmp = newSVsv(ST(i)); + U32 dualvar_flags = SvFLAGS(tmp) & ( SVf_IOK | SVf_NOK ); SV *stmp = S_sharedsv_new_shared(aTHX_ tmp); sharedsv_scalar_store(aTHX_ tmp, stmp); SHARED_CONTEXT; + if( SvPOK( stmp) ) { SvFLAGS(stmp) |= dualvar_flags; } av_store((AV*) sobj, i - 1, stmp); SvREFCNT_inc_void(stmp); CALLER_CONTEXT; ################################################################ ## Test set below, use same header as t/sv_simple.t ################################################################ use constant FOURTYTWO => dualvar(42,'Fourty-Two'); use constant PI => dualvar(3.14,'PI'); # This tests the following operations on shared arrays: # PUSH, UNSHIFT, POP, SHIFT (others are not significant as they won't take an SV as input) ## Test POP my @sa2 : shared = ( FOURTYTWO, FOURTYTWO ); ok(2, $sa2[0] == FOURTYTWO && $sa2[0] eq FOURTYTWO, 'SA2[0] contains a dualvar'); ok(3, $sa2[1] == FOURTYTWO && $sa2[1] eq FOURTYTWO, 'SA2[1] contains a dualvar'); ok(4, scalar @sa2 == 2 , 'SA2 is 2 elements long'); my $dv = pop @sa2; ok(5, $dv == FOURTYTWO, 'IV preserved after pop'); ok(6, $dv eq FOURTYTWO, 'PV preserved after pop'); ## Test SHIFT my $dv2 = shift @sa2; ok(7, $dv2 == FOURTYTWO, 'IV preserved after shift'); ok(8, $dv2 eq FOURTYTWO, 'PV preserved after shift'); ## Test PUSH my @sa : shared; push @sa , FOURTYTWO; ok(9, $sa[0] == FOURTYTWO, 'IV preserved after push'); ok(10, $sa[0] eq FOURTYTWO, 'PV preserved after push'); ## Test UNSHIFT unshift @sa, PI; ok(11, $sa[1] == FOURTYTWO && $sa[1] eq FOURTYTWO , 'Dualvar FOURTYTWO now in slot 1'); ok(12, $sa[0] == PI, 'PI.NV preserved after unshift'); ok(13, $sa[0] eq PI, 'PI.PV preserved after unshift'); ## Test shared-array to shared-array transfer. @sa3 should contain the same stuff my @sa3 : shared = @sa; ok(14, $sa3[0] == PI && $sa3[0] eq PI && $sa3[1] == FOURTYTWO && $sa3[1] eq FOURTYTWO, 'Copy of shared array ok'); #---------------------------------------------------------------------- # # Test operations on hashes. # Note that using a dualvar as hash key doesn't make sense, so we only test the value part. # my %sh : shared = ( pi => PI, fourtytwo => FOURTYTWO ); ok (15, (2 == scalar grep { $_ >0 && length $_ > 0 } values %sh) , 'VALUES returns 2 dualvars'); my $elem = delete $sh{ pi }; ok(16, $elem == PI && $elem eq PI, 'DELETE returns a dualvar'); # Test access via EACH (we have only one element, since previous test has deleted 'pi') ok( 17, 1 == scalar keys %sh, 'One key left in hash'); my ($k,$v) = each %sh; ok(18, $k eq 'fourtytwo', 'Key is "fourtytwo"'); ok(19, $v == FOURTYTWO && $v eq FOURTYTWO, 'Dualvar present'); exit(0); # EOF
Version 1.42 uploaded to CPAN.