Skip Menu |

This queue is for tickets about the RPC-XML CPAN distribution.

Report information
The Basics
Id: 56921
Status: open
Priority: 0/
Queue: RPC-XML

People
Owner: rjray [...] blackperl.com
Requestors: cpan [...] bargsten.org
Cc:
AdminCc:

Bug Information
Severity: Normal
Broken in: (no value)
Fixed in: (no value)



Subject: smart_encode in RPC/XML.pm loses elements, when element reference used twice or more
Using the same reference twice in a datastructure, smart_encode is not able to encode it properly. I attached a test, to make this clearer. Hope this helps, cheers joachim
Subject: same_reference.t
use warnings; use strict; use Test::More qw/no_plan/; BEGIN { use_ok('RPC::XML', qw/smart_encode/); } my $array_ref = [ "aa", "bb", "cc" ]; my $data1 = { a => $array_ref, b => $array_ref, }; my $data2 = { a => [ @{$array_ref} ], b => [ @{$array_ref} ], }; my $encoded_data1 = smart_encode($data1); my $encoded_data2 = smart_encode($data2); is_deeply($encoded_data1, $encoded_data2, "use same reference in data structure twice or more"); 1;
This is a tough one to approach. I specifically keep track of references that I've seen, because of a previous bug in which circular references were causing infinite recursion loops. I may not be able to fix this, I don't know. It may be that all I can do is put some cautionary documentation in the POD for smart_encode that warns you to copy your structures instead of trying to re-use them. -- """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" """""" Randy J. Ray Silicon Valley Scale Modelers: http://www.svsm.org rjray@blackperl.com randy.j.ray@gmail.com
Indeed, this is not easy to solve. In addition to an enhanced documentation, one could also add a 'smart_encode_weak(\@args, \@weak_references)' function that skips the reference counting for references in @weak_references. However, this is perhaps a bit too much for such a small bug. Am So 23. Jan 2011, 15:35:43, RJRAY schrieb: Show quoted text
> This is a tough one to approach. I specifically keep track of
references Show quoted text
> that I've seen, because of a previous bug in which circular
references Show quoted text
> were causing infinite recursion loops. > > I may not be able to fix this, I don't know. It may be that all I can
do Show quoted text
> is put some cautionary documentation in the POD for smart_encode that > warns you to copy your structures instead of trying to re-use them.
e.g. something like this...
Subject: smart_encode_weak.patch
diff -u -x .yo -x .git -x CVS -x .svn -x '*~' -x '.*.swp' -Nr RPC-XML-0.74/lib/RPC/XML.pm RPC-XML-0.74_jwb/lib/RPC/XML.pm --- RPC-XML-0.74/lib/RPC/XML.pm 2011-01-22 21:59:45.000000000 +0100 +++ RPC-XML-0.74_jwb/lib/RPC/XML.pm 2011-01-26 00:00:26.322306185 +0100 @@ -27,7 +27,7 @@ use warnings; use vars qw(@EXPORT_OK %EXPORT_TAGS $VERSION $ERROR %XMLMAP $XMLRE $ENCODING $FORCE_STRING_ENCODING $ALLOW_NIL); -use subs qw(time2iso8601 smart_encode utf8_downgrade); +use subs qw(time2iso8601 smart_encode smart_encode_weak utf8_downgrade); use base 'Exporter'; use Scalar::Util qw(blessed reftype); @@ -53,7 +53,7 @@ \&utf8::downgrade : sub { }; } -@EXPORT_OK = qw(time2iso8601 smart_encode +@EXPORT_OK = qw(time2iso8601 smart_encode smart_encode_weak RPC_BOOLEAN RPC_INT RPC_I4 RPC_DOUBLE RPC_DATETIME_ISO8601 RPC_BASE64 RPC_STRING RPC_NIL $ENCODING $FORCE_STRING_ENCODING $ALLOW_NIL); @@ -148,18 +148,29 @@ my $MAX_DOUBLE = 1e37; my $MIN_DOUBLE = $MAX_DOUBLE * -1; - sub smart_encode ## no critic (ProhibitExcessComplexity) + sub smart_encode_weak ## no critic (ProhibitExcessComplexity) { - my @values = @_; + my ($values, $weak_refs) = @_; my ($type, $seenrefs, @newvalues); + #take care of weak references + my $weak; + #seems to be a recursive call, weak refs is an existing hash + if(ref $weak_refs eq 'HASH') { + $weak = $weak_refs; + } + #api call, create a hash for easy checking + elsif( ref $weak_refs eq 'ARRAY') { + $weak = { map { $_ => 1} @{$weak_refs} }; + } + # Look for sooper-sekrit pseudo-blessed hashref as first argument. # It means this is a recursive call, and it contains a map of any # references we've already seen. - if ((blessed $values[0]) && ($values[0]->isa('RPC::XML::refmap'))) + if ((blessed $values->[0]) && ($values->[0]->isa('RPC::XML::refmap'))) { # Peel it off of the list - $seenrefs = shift @values; + $seenrefs = shift @{$values}; } else { @@ -167,7 +178,7 @@ $seenrefs = bless {}, 'RPC::XML::refmap'; } - foreach (@values) + foreach (@{$values}) { if (! defined $_) ## no critic (ProhibitCascadingIfElse) { @@ -177,7 +188,7 @@ elsif (ref $_) { # Skip any that we've already seen - next if $seenrefs->{$_}++; + next if( $seenrefs->{$_}++ && !exists($weak->{$_})); if (blessed($_) && ## no critic (ProhibitCascadingIfElse) ($_->isa('RPC::XML::datatype') || $_->isa('DateTime'))) @@ -210,7 +221,7 @@ # test be true even if the return value is a hard # undef. Only if the return value is an empty list # should this evaluate as false... - if (my @value = smart_encode($seenrefs, $_->{$key})) + if (my @value = smart_encode_weak([$seenrefs, $_->{$key}], $weak)) { $newhash{$key} = $value[0]; } @@ -227,7 +238,7 @@ # (see RT 35106) # Per RT 41063, looks like I get to deref $_ after all... $type = RPC::XML::array->new( - from => [ smart_encode($seenrefs, @{$_}) ] + from => [ smart_encode_weak([$seenrefs, @{$_}], $weak) ] ); } elsif (reftype($_) eq 'SCALAR') @@ -235,7 +246,7 @@ # This is a rare excursion into recursion, since the scalar # nature (de-refed from the object, so no longer magic) # will prevent further recursing. - $type = smart_encode($seenrefs, ${$_}); + $type = smart_encode_weak([$seenrefs, ${$_}], $weak); } else { @@ -301,6 +312,8 @@ return (wantarray ? @newvalues : $newvalues[0]); } + + sub smart_encode { return smart_encode_weak([ @_ ]); } } # This is a (mostly) empty class used as a common superclass for simple and diff -u -x .yo -x .git -x CVS -x .svn -x '*~' -x '.*.swp' -Nr RPC-XML-0.74/t/36_same_reference.t RPC-XML-0.74_jwb/t/36_same_reference.t --- RPC-XML-0.74/t/36_same_reference.t 1970-01-01 01:00:00.000000000 +0100 +++ RPC-XML-0.74_jwb/t/36_same_reference.t 2011-01-26 00:04:05.650965670 +0100 @@ -0,0 +1,23 @@ +use warnings; +use strict; +use Test::More qw/no_plan/; + +BEGIN { use_ok('RPC::XML', qw/smart_encode_weak/); } + +my $array_ref = [ "aa", "bb", "cc" ]; + +my $data1 = { + a => $array_ref, + b => $array_ref, +}; + +my $data2 = { + a => [ @{$array_ref} ], + b => [ @{$array_ref} ], +}; + +my $encoded_data1 = smart_encode_weak([$data1],[$array_ref]); +my $encoded_data2 = smart_encode_weak([$data2]); +is_deeply($encoded_data1, $encoded_data2, "use same reference in data structure twice or more with weak"); + +1;
On Tue Jan 25 18:08:56 2011, jwb wrote: Show quoted text
> e.g. something like this...
This patch doesn't work in my test case. This still loses an array element: use RPC::XML qw(smart_encode); use Data::Dumper; $Data::Dumper::Sortkeys = 1; use DateTime; my $x = DateTime->now(); my $y = $x->clone(); print Dumper(smart_encode([$x, $y]));
On Sun Jan 23 15:35:43 2011, RJRAY wrote: Show quoted text
> This is a tough one to approach. I specifically keep track of
references Show quoted text
> that I've seen, because of a previous bug in which circular references > were causing infinite recursion loops. > > I may not be able to fix this, I don't know. It may be that all I can
do Show quoted text
> is put some cautionary documentation in the POD for smart_encode that > warns you to copy your structures instead of trying to re-use them.
Maybe one of the serialization/unserialization modules can offer some inspiration? Martijn
On Thu Feb 24 09:27:58 2011, MSTREEK wrote: Show quoted text
> On Sun Jan 23 15:35:43 2011, RJRAY wrote:
> > This is a tough one to approach. I specifically keep track of
> references
> > that I've seen, because of a previous bug in which circular
references Show quoted text
> > were causing infinite recursion loops. > > > > I may not be able to fix this, I don't know. It may be that all I
can do Show quoted text
> > is put some cautionary documentation in the POD for smart_encode
that Show quoted text
> > warns you to copy your structures instead of trying to re-use them.
> > Maybe one of the serialization/unserialization modules can offer some > inspiration? > > Martijn
Hi Martijn, the correct usage would be: --------------------------------8<---------------------------------------------- use RPC::XML qw(smart_encode smart_encode_weak); =head2 smart_encode_weak([@args], [@list_of_weak_references]) Does the same as C<smart_encode(@args)> but with the difference that all references in @list_of_weak_references are not squeezed. =cut use Data::Dumper; $Data::Dumper::Sortkeys = 1; use DateTime; my $x = DateTime->now(); my $y = $x->clone(); #does not work properly print Dumper(smart_encode([$x, $y])); #the modified implementation that works (note the additional []) print Dumper(smart_encode_weak([[$x, $y]], [$x, $y])); --------------------------------8<---------------------------------------------- I added smart_encode_weak to handle such cases. Of course you need to know the references in question in advance. Best regards, Joachim
On Thu Feb 24 10:08:59 2011, jwb wrote: Show quoted text
> I added smart_encode_weak to handle such cases. Of course you need to > know the > references in question in advance.
That means I'd have to walk my data structure, find all DateTime objects, add them to the "weak" list, and then have smart_encode handle it. That feels like a lot of double work: if my code walks the data structure to get all DateTime instances anyway, it feels like I should also be doing smart_encode work at the same time (converting to RPC::XML::datetime_iso8601). The "not skipping double references" works fine for me, in my opinion the warning about circular references in the documentation is the best solution (Perl will complain about deep recursion in that case anyway). Martijn
On Fri Feb 25 04:54:37 2011, MSTREEK wrote: Show quoted text
> On Thu Feb 24 10:08:59 2011, jwb wrote: >
> > I added smart_encode_weak to handle such cases. Of course you need
to Show quoted text
> > know the > > references in question in advance.
> > That means I'd have to walk my data structure, find all DateTime > objects, add them to the "weak" list, and then have smart_encode
handle Show quoted text
> it. > > That feels like a lot of double work: if my code walks the data > structure to get all DateTime instances anyway, it feels like I
should Show quoted text
> also be doing smart_encode work at the same time (converting to > RPC::XML::datetime_iso8601). > > The "not skipping double references" works fine for me, in my opinion > the warning about circular references in the documentation is the
best Show quoted text
> solution (Perl will complain about deep recursion in that case
anyway). Show quoted text
> > Martijn
Indeed, this is not the best solution, but I think it is better to keep smart_encode as is, since a lot of people already rely on the 'no circular references' behaviour. So, feel free to write a wrapper for smart_encode_weak or, perhaps you can find a better solution. Serialization modules, e.g. YAML or Storable, don't have this problem, since they can store the reference 'as is'. Due to the lack of solutions, the current one works for me... Joachim
I think I have a solution for this. I will be trying it this week, and if it works it will be fixed in 0.77. -- Randy J. Ray rjray@blackperl.com randy.j.ray@gmail.com
From: des [...] des.no
I believe this is the same loop detection bug that was fixed in 0.78. I verified that Joachim's test fails with 0.77 and succeeds with 0.78.