Skip Menu |

This queue is for tickets about the Compress-LZ4 CPAN distribution.

Report information
The Basics
Id: 75624
Status: resolved
Priority: 0/
Queue: Compress-LZ4

People
Owner: Nobody in particular
Requestors: danielqzinn [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



CC: gray [...] cpan.org
Subject: decompress(compress($x)) does not equal $x
Date: Thu, 08 Mar 2012 10:58:14 -0800
To: bug-Compress-LZ4 [...] rt.cpan.org
From: Daniel Zinn <danielqzinn [...] gmail.com>
Dear Sir or Madam: Please find attached an example perl script to reproduce the bug. I am using the current version of Compress::LZ4 from CPAN and the tests that come with it went through fine. More information about my environment is attached. I also attached a .t file that can be used to reproduce the test as part of the distribution. Thank you! Daniel

Message body is not shown because sender requested not to inline it.

Message body is not shown because sender requested not to inline it.

Message body is not shown because sender requested not to inline it.

Message body is not shown because sender requested not to inline it.

On Thu Mar 08 13:58:27 2012, danielqzinn@gmail.com wrote: Show quoted text
> Please find attached an example perl script to reproduce the bug. > > I am using the current version of Compress::LZ4 from CPAN and the tests > that come with it went through fine. More information about my > environment is attached. > > I also attached a .t file that can be used to reproduce the test as part > of the distribution.
Please verify that this is not a bug in the lz4 c code: svn checkout http://lz4.googlecode.com/svn/trunk/ lz4 cd lz4 make ./lz4demo64.exe -c $input_file output ./lz4demo64.exe -t output ./lz4demo64.exe -d roundtrip $input_file diff output roundtrip ./lz4demo32.exe -c $input_file output ./lz4demo32.exe -t output ./lz4demo32.exe -d roundtrip $input_file diff output roundtrip If anything above fails, it's a problem with the c code and you should report it to that project. If the above succeeded, then I'm not sure what the problem is.
CC: danielqzinn [...] gmail.com
Subject: Re: [rt.cpan.org #75624] decompress(compress($x)) does not equal $x
Date: Thu, 08 Mar 2012 14:36:31 -0800
To: bug-Compress-LZ4 [...] rt.cpan.org
From: Daniel Zinn <danielqzinn [...] gmail.com>
The LZ4 binary performs a correct round-trip when run on the file containing the data that fails the roundtrip under Perl (see first block). Best, Daniel PS: When compressing the file with the simple attached Perl program (compress.pl), then it produces a different compressed string than what is inside the file the lz4 binary produces (see second block) -- is that to be expected? Cheers, Daniel 8<----------------------------------------------------------------------- q@qew:~/tmp/lz4$ cat test.pl #!/usr/bin/perl use Compress::LZ4; my $in = `cat $0`; my $dest = decompress(compress($in)); print "=" x 80 ."\n" . $in; print "=" x 80 ."\n" . $dest; print "=" x 80 . "\n"; die unless $in eq $dest; q@qew:~/tmp/lz4$ ./lz4demo64.exe -c test.pl test.pl.compressed *** Compression CLI using LZ4 algorithm , by Yann Collet (Mar 8 2012) *** Compressed 199 bytes into 157 bytes ==> 78.89% Done in 0.00 s ==> inf MB/s q@qew:~/tmp/lz4$ ./lz4demo64.exe -t test.pl.compressed *** Compression CLI using LZ4 algorithm , by Yann Collet (Mar 8 2012) *** Successfully decoded 199 bytes Done in 0.00 s ==> inf MB/s q@qew:~/tmp/lz4$ ./lz4demo64.exe -d test.pl.compressed test.pl.roundtrip *** Compression CLI using LZ4 algorithm , by Yann Collet (Mar 8 2012) *** Successfully decoded 199 bytes Done in 0.00 s ==> inf MB/s q@qew:~/tmp/lz4$ diff test.pl test.pl.roundtrip q@qew:~/tmp/lz4$ 8<----------------------------------------------------------------------- q@qew:~/tmp/lz4$ ./compress.pl test.pl > test.pl.compressed_Perl q@qew:~/tmp/lz4$ diff test.pl.compressed test.pl.compressed_Perl Binary files test.pl.compressed and test.pl.compressed_Perl differ q@qew:~/tmp/lz4$ ./lz4demo64.exe -t test.pl.compressed_Perl *** Compression CLI using LZ4 algorithm , by Yann Collet (Mar 8 2012) *** Unrecognized header : file cannot be decoded q@qew:~/tmp/lz4$ ls -la test.pl.compressed* -rw-rw-r-- 1 q q 157 Mar 8 14:27 test.pl.compressed -rw-rw-r-- 1 q q 168 Mar 8 14:28 test.pl.compressed_Perl On 03/08/2012 02:16 PM, gray via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=75624 > > > On Thu Mar 08 13:58:27 2012, danielqzinn@gmail.com wrote: >
>> Please find attached an example perl script to reproduce the bug. >> >> I am using the current version of Compress::LZ4 from CPAN and the tests >> that come with it went through fine. More information about my >> environment is attached. >> >> I also attached a .t file that can be used to reproduce the test as part >> of the distribution.
> > Please verify that this is not a bug in the lz4 c code: > svn checkout http://lz4.googlecode.com/svn/trunk/ lz4 > cd lz4 > make > ./lz4demo64.exe -c $input_file output > ./lz4demo64.exe -t output > ./lz4demo64.exe -d roundtrip $input_file > diff output roundtrip > > ./lz4demo32.exe -c $input_file output > ./lz4demo32.exe -t output > ./lz4demo32.exe -d roundtrip $input_file > diff output roundtrip > > If anything above fails, it's a problem with the c code and you should > report it to that project. If the above succeeded, then I'm not sure > what the problem is.

Message body is not shown because sender requested not to inline it.

On Thu Mar 08 17:36:43 2012, danielqzinn@gmail.com wrote: Show quoted text
> PS: When compressing the file with the simple attached Perl program > (compress.pl), then it produces a different compressed string than what > is inside the file the lz4 binary produces (see second block) -- is that > to be expected?
Yes. The lz4 compression function does not store the length of the resulting compressed data in the data itself, so the perl interface must do that, since the decompression must know the size of the original data.
Subject: Re: [rt.cpan.org #75624] decompress(compress($x)) does not equal $x
Date: Thu, 08 Mar 2012 19:46:48 -0800
To: bug-Compress-LZ4 [...] rt.cpan.org
From: Daniel Zinn <danielqzinn [...] gmail.com>
Show quoted text
> Yes. The lz4 compression function does not store the length of the > resulting compressed data in the data itself, so the perl interface must > do that, since the decompression must know the size of the original data.
Well, then I don't have an idea of why lz4 works in general, but not when called via Compress::LZ4.
On Thu Mar 08 22:46:58 2012, danielqzinn@gmail.com wrote: Show quoted text
> > Yes. The lz4 compression function does not store the length of the > > resulting compressed data in the data itself, so the perl interface must > > do that, since the decompression must know the size of the original
data. Show quoted text
> > Well, then I don't have an idea of why lz4 works in general, but not > when called via Compress::LZ4.
It might be a case of pointer truncation - sizeof(int) != sizeof(void*). I'm going to try emulating x86_64 with qemu and see if I can duplicate the problem.
Subject: Re: [rt.cpan.org #75624] decompress(compress($x)) does not equal $x
Date: Thu, 08 Mar 2012 22:46:56 -0800
To: bug-Compress-LZ4 [...] rt.cpan.org
From: Daniel Zinn <danielqzinn [...] gmail.com>
Show quoted text
> It might be a case of pointer truncation - sizeof(int) != sizeof(void*). > I'm going to try emulating x86_64 with qemu and see if I can duplicate > the problem.
I can confirm that the bug does not appear on a 32-bit machine. Cheers, daniel With latest version from CPAN: q@polaris:~$ uname -a Linux polaris 2.6.32-2-686-bigmem #1 SMP Thu Feb 11 04:55:32 UTC 2010 i686 GNU/Linux q@polaris:~$ ./test.pl ================================================================================ #!/usr/bin/perl use Compress::LZ4; my $in = `cat $0`; my $dest = decompress(compress($in)); print "=" x 80 ."\n" . $in; print "=" x 80 ."\n" . $dest; print "=" x 80 . "\n"; die unless $in eq $dest; ================================================================================ #!/usr/bin/perl use Compress::LZ4; my $in = `cat $0`; my $dest = decompress(compress($in)); print "=" x 80 ."\n" . $in; print "=" x 80 ."\n" . $dest; print "=" x 80 . "\n"; die unless $in eq $dest; ================================================================================ q@polaris:~$ perl -v This is perl 5, version 12, subversion 4 (v5.12.4) built for i486-linux-gnu-thread-multi-64int (with 46 registered patches, see perl -V for more detail) Copyright 1987-2010, Larry Wall Perl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Perl 5 source kit. Complete documentation for Perl, including FAQ lists, should be found on this system using "man perl" or "perldoc perl". If you have access to the Internet, point your browser at http://www.perl.org/, the Perl Home Page. q@polaris:~$
On Fri Mar 09 01:42:21 2012, GRAY wrote: Show quoted text
> It might be a case of pointer truncation - sizeof(int) != sizeof(void*). > I'm going to try emulating x86_64 with qemu and see if I can duplicate > the problem.
I was able to reproduce this bug with VirtualBox, but I can't narrow down the cause. Neither the python interface or the plain C interface exhibited the problem, so it's definitely on the perl side. But it must be something really obscure or really obvious...
fixed in 0.04
Subject: Re: [rt.cpan.org #75624] x86_64: decompress(compress($x)) does not equal $x
Date: Mon, 19 Mar 2012 22:42:06 -0700
To: bug-Compress-LZ4 [...] rt.cpan.org
From: Daniel Zinn <danielqzinn [...] gmail.com>
I can confirm that the problem is fixed. Thank you! The only (relevant?) change I see on github is that you removed src_len from computing the dest_len. Is it easy to explain what the problem was? And how you fixed it? Cheers, Daniel
On Tue Mar 20 01:42:19 2012, danielqzinn@gmail.com wrote: Show quoted text
> I can confirm that the problem is fixed. Thank you! The only (relevant?) > change I see on github is that you removed src_len from computing the > dest_len. Is it easy to explain what the problem was? And how you
fixed it? There is an unknown bug somewhere that didn't like the amalgamated/unity build method, so I switched to compiling each c file to an object and linking together. Look in the Makefile.PL if you're interested.