Bug #122695 for Compress-Raw-Zlib: sync flush appending extra empty uncompressed block

Wed Aug 02 20:40:51 2017 felipe [...] felipegasper.com - Ticket created

Subject:	sync flush appending extra empty uncompressed block
Date:	Thu, 3 Aug 2017 01:48:09 +0200
To:	bug-Compress-Raw-Zlib [...] rt.cpan.org
From:	Felipe Gasper <felipe [...] felipegasper.com>

#!/usr/bin/env perl use strict; use warnings; use Compress::Raw::Zlib; my $deflate = Compress::Raw::Zlib::Deflate->new( -WindowBits => -15, -AppendOutput => 1 ); my $str = 'Hello'; for ( 1 .. 2 ) { $deflate->deflate( $str, my $out ); $deflate->flush( $out, Compress::Raw::Zlib::Z_SYNC_FLUSH() ); printf "sync: %v.02x\n", $out; $out = q<>; $deflate->deflate( $str, $out ); $deflate->flush( $out, Compress::Raw::Zlib::Z_FULL_FLUSH() ); printf "full: %v.02x\n", $out; } =================== sync: f2.48.cd.c9.c9.07.00.00.00.ff.ff.00.00.00.ff.ff full: f2.00.11.00.00.00.00.ff.ff sync: f2.48.cd.c9.c9.07.00.00.00.ff.ff full: f2.00.11.00.00.00.00.ff.ff =================== That first sync doesn’t seem like it should be appending the extra 5 bytes? -FG

Sun Aug 06 04:47:50 2017 pmqs [...] cpan.org - Correspondence added

Hey Filipe, my reading of the zlib docs (https://zlib.net/manual.html) says that is expected behaviour for Z_SYNC_FLUSH. Below is the relevant paragraph from the "deflate" section that talks about the extra block appended by Z_SYNC_FLUSH. If the parameter flush is set to Z_SYNC_FLUSH, all pending output is flushed to the output buffer and the output is aligned on a byte boundary, so that the decompressor can get all input data available so far. (In particular avail_in is zero after the call if enough output space has been provided before the call.) Flushing may degrade compression for some compression algorithms and so it should be used only when necessary. This completes the current deflate block and follows it with an empty stored block that is three bits plus filler bits to the next byte, followed by four bytes (00 00 ff ff).

Sun Aug 06 04:47:51 2017 The RT System itself - Status changed from 'new' to 'open'

Sun Aug 06 06:31:16 2017 felipe [...] felipegasper.com - Correspondence added

Subject:	Re: [rt.cpan.org #122695] sync flush appending extra empty uncompressed block
Date:	Sun, 6 Aug 2017 12:30:26 +0200
To:	bug-Compress-Raw-Zlib [...] rt.cpan.org
From:	Felipe Gasper <felipe [...] felipegasper.com>

Show quoted text

> On Aug 6, 2017, at 10:47 AM, Paul Marquess via RT <bug-Compress-Raw-Zlib@rt.cpan.org> wrote: > > <URL: https://rt.cpan.org/Ticket/Display.html?id=122695 > > > Hey Filipe, > > my reading of the zlib docs (https://zlib.net/manual.html) says that is expected behaviour for Z_SYNC_FLUSH. Below is the relevant paragraph from the "deflate" section that talks about the extra block appended by Z_SYNC_FLUSH. > > > If the parameter flush is set to Z_SYNC_FLUSH, all pending output is flushed to the output buffer and the output is aligned on a byte boundary, so that the decompressor can get all input data available so far. (In particular avail_in is zero after the call if enough output space has been provided before the call.) Flushing may degrade compression for some compression algorithms and so it should be used only when necessary. This completes the current deflate block and follows it with an empty stored block that is three bits plus filler bits to the next byte, followed by four bytes (00 00 ff ff). >

Hi Paul, C::R::Z, though, is appending *two* empty blocks with 00.00.ff.ff -- and the behaviour is different the first time from the second time. Should the two Z_SYNC_FLUSH calls not produce the same output given that there is a Z_FULL_FLUSH between them? Thank you! -FG

Sun Aug 06 14:21:00 2017 pmqs [...] cpan.org - Correspondence added

On Sun Aug 06 06:31:16 2017, felipe@felipegasper.com wrote: Show quoted text

> > On Aug 6, 2017, at 10:47 AM, Paul Marquess via RT <bug-Compress-Raw- > > Zlib@rt.cpan.org> wrote: > > > > <URL: https://rt.cpan.org/Ticket/Display.html?id=122695 > > > > > Hey Filipe, > > > > my reading of the zlib docs (https://zlib.net/manual.html) says that > > is expected behaviour for Z_SYNC_FLUSH. Below is the relevant > > paragraph from the "deflate" section that talks about the extra block > > appended by Z_SYNC_FLUSH. > > > > > > If the parameter flush is set to Z_SYNC_FLUSH, all pending output is > > flushed to the output buffer and the output is aligned on a byte > > boundary, so that the decompressor can get all input data available > > so far. (In particular avail_in is zero after the call if enough > > output space has been provided before the call.) Flushing may degrade > > compression for some compression algorithms and so it should be used > > only when necessary. This completes the current deflate block and > > follows it with an empty stored block that is three bits plus filler > > bits to the next byte, followed by four bytes (00 00 ff ff). > >

> > Hi Paul, > > C::R::Z, though, is appending *two* empty blocks with 00.00.ff.ff -- > and the behaviour is different the first time from the second time. > > Should the two Z_SYNC_FLUSH calls not produce the same output given > that there is a Z_FULL_FLUSH between them? > > Thank you! > > -FG

Thanks for coming back to me on this Filipe - that does indeed look like an extra block sneaking in. Apologies for not spotting it first time. The thing is, the code I use to interface to zlib should be doing *exactly* the same thing both time the call with Z_SYNC_FLUSH is made. Unless this is a bug in zlib (which is very unlikely), there is something slightly sub-optimal with how I'm interfacing to zlib. Need to dig into the code & see what is triggering the extra block. cheers Paul