| Subject: | Potential problem with binary strings an Queue::Q::ReliableFIFO::Redis |
| Date: | Sun, 20 Apr 2014 20:22:40 -0400 |
| To: | bug-Queue-Q [...] rt.cpan.org |
| From: | Philip Gladstone <philip [...] gladstonefamily.net> |
I just installed Queue::Q from CPAN.
Version 0.23
On the enqueuing side:
while (my $otheraddr = $sock->recv($pkt, 4096)) {
$q->enqueue_item({ remaddr => $otheraddr, rxtime => time(), packet
=> $pkt });
}
In particular, this is an IPv4 socket, so the $otheraddr is always 16 bytes.
On the dequeue side:
$item = $q->claim_item();
my $data = $item->data;
my $remaddr = $data->{remaddr};
Now, $remaddr is no longer 16 bytes.
From doing some experimenting, it may be a UTF8 issue with JSON::XS.
The following test program shows the problem:
#!/usr/bin/perl
#
use strict;
use JSON::XS;
use Data::Dumper;
my $a =
"{\"b\":{\"remaddr\":\"\\u0002\\u0000\\u00042\x{c3}\x{8f}vj\@\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\\u0000\"},\"t\":1397921344}";
my $b = decode_json $a;
my $d = $b->{b};
my $b2 = JSON::XS->new->pretty(0)->decode($a);
my $d2 = $b2->{b};
print Dumper($d, length($d->{remaddr}));
print Dumper($d2, length($d2->{remaddr}));
This generates the output:
$VAR1 = {
'remaddr' => "2\x{cf}vj\@"
};
$VAR2 = 16;
$VAR1 = {
'remaddr' => "2\x{c3}\x{8f}vj\@"
};
$VAR2 = 17;
The utf8 version of the decoder generates something that is the right
length, while the non-utf8 version does not (and this is the version
used in Queue::Q::ReliableFIFO::Item).
I am not sufficiently versed in the mysteries of perl and utf8 to know
what the issue is.
If you add the following block to the end of test_claim_fifo (in
t/lib/Queue/Q/TestReliableFIFO.pm
my $a = pack("C*", 28, 29, 30);
$q->enqueue_item($a);
$item = $q->claim_item();
$q->mark_item_as_done($item);
is($item->data, $a, "verify roundtrip: " . join(',', unpack("C*",
$a)));
$a = pack("C*", 128, 129, 130);
$q->enqueue_item($a);
$item = $q->claim_item();
$q->mark_item_as_done($item);
is($item->data, $a, "verify roundtrip: " . join(',', unpack("C*",
$a)));
then the second test will fail (and the first succeed) (assuming that
you have a redis server to work with).
Philip