Skip Menu |

This queue is for tickets about the Compress-Raw-Zlib CPAN distribution.

Report information
The Basics
Id: 119762
Status: resolved
Priority: 0/
Queue: Compress-Raw-Zlib

People
Owner: Nobody in particular
Requestors: vcunat [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in:
  • 2.065
  • 2.071
Fixed in: 2.072



Subject: Tests broken with zlib-1.2.10
I haven't looked into why the tests started failing. It might be just a problem in the tests themselves. You can grab a full log on http://hydra.nixos.org/build/46157592/nixlog/4/raw
Thanks, known issue. zlib-1.2.10 changed the API for the deflateParams function. Know how to fix it, just haven't got the fix completed yet. Paul
On Sun Jan 08 08:45:54 2017, PMQS wrote: Show quoted text
> zlib-1.2.10 changed the API for the deflateParams function. [...]
So, I assume that means the current version won't (fully) work with zlib-1.2.10...
On Sun Jan 08 12:07:10 2017, https://vcunat.mojeid.cz/#53tSKthUzV wrote: Show quoted text
> On Sun Jan 08 08:45:54 2017, PMQS wrote:
> > zlib-1.2.10 changed the API for the deflateParams function. [...]
> > So, I assume that means the current version won't (fully) work with > zlib-1.2.10...
Correct. The function not working (deflateParams) is one I doubt many (if any) folk rely on. The fix is proving problematic, so may just temporarily remove support for deflateParams if I don't get it sorted soon.
Side note: I suppose IO::Compress::Gzip tests suffer from the very same problem.
On Tue Jan 10 08:33:12 2017, https://vcunat.mojeid.cz/#53tSKthUzV wrote: Show quoted text
> Side note: I suppose IO::Compress::Gzip tests suffer from the very > same problem.
No, they won't. Assuming the deflateParams issue get fixed in Compress::Raw::Zlib, there would be no impact on the IO:Compress::* modules at all. Reason is very straightforward -- IO:Compress does not use deflateParams.
On Tue Jan 10 09:28:31 2017, PMQS wrote: Show quoted text
> On Tue Jan 10 08:33:12 2017, https://vcunat.mojeid.cz/#53tSKthUzV > wrote:
> > Side note: I suppose IO::Compress::Gzip tests suffer from the very > > same problem.
> > No, they won't. Assuming the deflateParams issue get fixed in > Compress::Raw::Zlib, there would be no impact on the IO:Compress::* > modules at all. > > Reason is very straightforward -- IO:Compress does not use > deflateParams.
I wrote that because I *am* getting new test errors in there after disabling tests in IO::Compress::Gzip (in hope that the broken stuff won't really be used anywhere).
On Tue Jan 10 09:37:16 2017, https://vcunat.mojeid.cz/#53tSKthUzV wrote: Show quoted text
> On Tue Jan 10 09:28:31 2017, PMQS wrote:
> > On Tue Jan 10 08:33:12 2017, https://vcunat.mojeid.cz/#53tSKthUzV > > wrote:
> > > Side note: I suppose IO::Compress::Gzip tests suffer from the very > > > same problem.
> > > > No, they won't. Assuming the deflateParams issue get fixed in > > Compress::Raw::Zlib, there would be no impact on the IO:Compress::* > > modules at all. > > > > Reason is very straightforward -- IO:Compress does not use > > deflateParams.
> > I wrote that because I *am* getting new test errors in there after > disabling tests in IO::Compress::Gzip (in hope that the broken stuff > won't really be used anywhere).
ok, thanks. Will try disabling the deflateParams tests in Compress::Raw::Zlib to get it building & see what happens with IO::Compress. Hope the zlib API hasn't changed somewhere else.
On Sun Jan 08 08:45:54 2017, PMQS wrote: Show quoted text
> Thanks, known issue. > > zlib-1.2.10 changed the API for the deflateParams function. Know how > to fix it, just haven't got the fix completed yet. > > Paul
Side note, for obvious reasons this is now also making the tests of Perl 5 fail.
On Fri Feb 03 10:07:19 2017, AKHUETTEL wrote: ... Hey Andreas, Show quoted text
> Side note, for obvious reasons this is now also making the tests of > Perl 5 fail.
Yes and no. :-) The standard perl distribution includes its own copy of the zlib source, and by default the build uses that. So the zlib-1.2.10 change won't impact that use-case. If you build with a pre-installed zlib, then, or course, it is a different story. Fix is in the works. Just waiting for the dust to settle on the new zlib release - there have been a flurry of updates.
On Fri Feb 03 10:51:52 2017, PMQS wrote: Show quoted text
> > The standard perl distribution includes its own copy of the zlib > source, and by default the build uses that. So the zlib-1.2.10 change > won't impact that use-case. >
Thanks. Offtopic here, but... that looks like 1.2.8 ... has it been patched for CVE-2016-9840, CVE-2016-9841, CVE-2016-9842, CVE-2016-9843 ? :/
From: vcunat [...] gmail.com
On Sun Feb 05 06:31:16 2017, AKHUETTEL wrote: Show quoted text
> Thanks. Offtopic here, but... that looks like 1.2.8 ... has it been > patched for CVE-2016-9840, CVE-2016-9841, CVE-2016-9842, CVE-2016-9843 > ? :/
The security update was the very thing that forced me to update zlib in distribution and discover the failing tests during CI (and disregard the failure for now).
On Sun Feb 05 06:31:16 2017, AKHUETTEL wrote: Show quoted text
> On Fri Feb 03 10:51:52 2017, PMQS wrote:
> > > > The standard perl distribution includes its own copy of the zlib > > source, and by default the build uses that. So the zlib-1.2.10 change > > won't impact that use-case. > >
> > Thanks. Offtopic here, but... that looks like 1.2.8 ... has it been > patched for CVE-2016-9840, CVE-2016-9841, CVE-2016-9842, CVE-2016-9843 > ? :/
The zlib source I ship will not have these addressed. I intend updating my distribution to ship with part of zlib 1.2.11 soon(ish). I see that upstream zlib has addresses those issue I'll get them for free. Paul
From: ppisar [...] redhat.com
Dne Ne 08.led.2017 08:45:54, PMQS napsal(a): Show quoted text
> Thanks, known issue. > > zlib-1.2.10 changed the API for the deflateParams function. Know how > to fix it, just haven't got the fix completed yet. >
I located the change in zlib-1.2.9 and after some difficulties, I implemented a change in the Compress-Raw-Zlib test. The tests pass with 1.2.8 as well as with 1.2.11. I had some difficulties to with a fix by retrying deflateParams(), so chose flushing a stream. Maybe there is some issue with Compress-Raw-Zlib how to redefines stream positions. Or it can a real bug in zlib. Also maybe you would like to add an implicit stream flush into Perl's deflateParams() to achieve better compatibility with existing Compress::Raw::Zlib users. I did not do that.
Subject: Compress-Raw-Zlib-2.071-Adapt-tests-to-zlib-1.2.11.patch
From b42f5c088158f473116d3aca2d050d4efb95b021 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Petr=20P=C3=ADsa=C5=99?= <ppisar@redhat.com> Date: Tue, 7 Feb 2017 14:44:48 +0100 Subject: [PATCH] Adapt tests to zlib 1.2.11 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Since zlib-1.2.9 it's not safe to invoke deflateParams() when not all input was consumed by deflate(). deflateParams() could return Z_BUF_ERROR in some cases: commit 7161ad76e2d0ac7de2a6235fcad3b9dfc99e9140 Author: Mark Adler <madler@alumni.caltech.edu> Date: Tue Nov 22 23:29:19 2016 -0800 Assure that deflateParams() will not switch functions mid-block. This alters the specification in zlib.h, so that deflateParams() will not change any parameters if there is not enough output space in the event that a block is emitted in order to allow switching the compression function. zlib documentation recommends two fixes: To retry deflateParams() as it processes a piece of input underneath until something else than Z_BUF_ERROR is returned. However this does not work for me because then the compressed stream gets corrupted and the subsequent inflate() returns a failure. Another fix is to flush the deflated stream with Z_BLOCK just before any deflateParams() call that follows unifinished deflate(). This assures the new deflate options will be applied immediatelly on next deflate() call. This fix works for me. Thus I implemented it in the tests. The new tests pass with zlib 1.2.8 as well as 1.2.11. 1.2.9 and 1.2.10 seems broken changing the deflate options was fixed in 1.2.11. CPAN RT#119762 Signed-off-by: Petr Písař <ppisar@redhat.com> --- t/02zlib.t | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/t/02zlib.t b/t/02zlib.t index 2c9aad6..50f6a82 100644 --- a/t/02zlib.t +++ b/t/02zlib.t @@ -24,13 +24,13 @@ BEGIN my $count = 0 ; if ($] < 5.005) { - $count = 232 ; + $count = 236 ; } elsif ($] >= 5.006) { - $count = 317 ; + $count = 320 ; } else { - $count = 275 ; + $count = 278 ; } plan tests => $count + $extra; @@ -537,6 +537,7 @@ SKIP: $status = $x->deflate($hello, $Answer) ; cmp_ok $status, '==', Z_OK ; + cmp_ok $x->flush($Answer, Z_BLOCK), '==', Z_OK ; $input .= $hello; # error cases @@ -561,6 +562,7 @@ SKIP: $status = $x->deflate($goodbye, $Answer) ; cmp_ok $status, '==', Z_OK ; + cmp_ok $x->flush($Answer, Z_BLOCK), '==', Z_OK ; $input .= $goodbye; # change only Level @@ -572,6 +574,7 @@ SKIP: $status = $x->deflate($goodbye, $Answer) ; cmp_ok $status, '==', Z_OK ; + cmp_ok $x->flush($Answer, Z_BLOCK), '==', Z_OK ; $input .= $goodbye; # change only Strategy -- 2.7.4
From: paul [...] city-fan.org
On Tue Feb 07 09:09:01 2017, ppisar wrote: Show quoted text
> Dne Ne 08.led.2017 08:45:54, PMQS napsal(a):
> > Thanks, known issue. > > > > zlib-1.2.10 changed the API for the deflateParams function. Know how > > to fix it, just haven't got the fix completed yet. > >
> I located the change in zlib-1.2.9 and after some difficulties, I > implemented a change in the Compress-Raw-Zlib test. The tests pass > with 1.2.8 as well as with 1.2.11.
Although the tests pass with 1.2.8, they fail with anything older, such as zlib 1.2.7 on RHEL-7.
On Tue Feb 07 10:10:17 2017, paul@city-fan.org wrote: Show quoted text
> On Tue Feb 07 09:09:01 2017, ppisar wrote:
> > Dne Ne 08.led.2017 08:45:54, PMQS napsal(a):
> > > Thanks, known issue. > > > > > > zlib-1.2.10 changed the API for the deflateParams function. Know > > > how > > > to fix it, just haven't got the fix completed yet. > > >
> > I located the change in zlib-1.2.9 and after some difficulties, I > > implemented a change in the Compress-Raw-Zlib test. The tests pass > > with 1.2.8 as well as with 1.2.11.
> > Although the tests pass with 1.2.8, they fail with anything older, > such as zlib 1.2.7 on RHEL-7.
Can you post a log of the failures please?
From: paul [...] city-fan.org
On Tue Feb 07 10:43:22 2017, PMQS wrote: Show quoted text
> On Tue Feb 07 10:10:17 2017, paul@city-fan.org wrote:
> > On Tue Feb 07 09:09:01 2017, ppisar wrote:
> > > Dne Ne 08.led.2017 08:45:54, PMQS napsal(a):
> > > > Thanks, known issue. > > > > > > > > zlib-1.2.10 changed the API for the deflateParams function. Know > > > > how > > > > to fix it, just haven't got the fix completed yet. > > > >
> > > I located the change in zlib-1.2.9 and after some difficulties, I > > > implemented a change in the Compress-Raw-Zlib test. The tests pass > > > with 1.2.8 as well as with 1.2.11.
> > > > Although the tests pass with 1.2.8, they fail with anything older, > > such as zlib 1.2.7 on RHEL-7.
> > Can you post a log of the failures please?
$ make test COMPRESS_ZLIB_RUN_MOST=1 PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/000prereq.t ...... ok t/01version.t ...... ok # Failed test (t/02zlib.t at line 558) # got: -5 # expected: 0 # Failed test (t/02zlib.t at line 570) # got: -5 # expected: 0 # Failed test (t/02zlib.t at line 582) # got: -5 # expected: 0 # Looks like you failed 3 tests of 321. t/02zlib.t ......... Dubious, test returned 3 (wstat 768, 0x300) Failed 3/321 subtests t/07bufsize.t ...... ok t/09limitoutput.t .. ok t/18lvalue.t ....... ok t/19nonpv.t ........ ok t/99pod.t .......... ok Test Summary Report ------------------- t/02zlib.t (Wstat: 768 Tests: 321 Failed: 3) Failed tests: 149, 154, 159 Non-zero exit status: 3 Files=8, Tests=774, 1 wallclock secs ( 0.04 usr 0.00 sys + 0.49 cusr 0.02 csys = 0.55 CPU) Result: FAIL Failed 1/8 test programs. 3/774 subtests failed. make: *** [test_dynamic] Error 255
From: paul [...] city-fan.org
With zlib 1.2.3 on RHEL-5 and RHEL-6, the "got" values are -2 rather than -5.
From: ppisar [...] redhat.com
Dne Út 07.úno.2017 09:09:01, ppisar napsal(a): Show quoted text
> I had some difficulties to with a fix by retrying deflateParams()
Now I know why. That's because _deflateParams() allows zlib's deflateParams() to store only one byte and appends it to the output variable on on flush() or deflate(). Any inbetween _deflateParams() call will discard the previously stored byte. This leads to the data corruption I saw when trying to exhaust not yet deflated input by repeated deflate() calls in the tests. I can see there is an alternative buffer implementation for the _deflateParams() temporary output, but it's disabled by the SETP_BYTE macro by default. Actually I doubt the alternative buffer implementation works. First it does not grow in _deflateParams(). Second the grow is wrong in flush() and deflate() because s->stream.next_out is not updated after calling Sv_Grow(). Sv_Grow() is sv_grow() and one should use sv_grow() return value for the purpose because the SV's character buffer can be moved.
On Wed Feb 08 10:27:38 2017, ppisar wrote: Show quoted text
> Dne Út 07.úno.2017 09:09:01, ppisar napsal(a):
> > I had some difficulties to with a fix by retrying deflateParams()
> > Now I know why. That's because _deflateParams() allows zlib's > deflateParams() to store only one byte and appends it to the output > variable on on flush() or deflate(). Any inbetween _deflateParams() > call will discard the previously stored byte. This leads to the data > corruption I saw when trying to exhaust not yet deflated input by > repeated deflate() calls in the tests. > > I can see there is an alternative buffer implementation for the > _deflateParams() temporary output, but it's disabled by the SETP_BYTE > macro by default. Actually I doubt the alternative buffer > implementation works.
The code enabled when SETP_BYTE is off most certainly does not work. It was only ever a work-in-progress that was never completed. The code enabled by SETP_BYTE made the interface to deflatParams a lot simpler and worked fine with zlib prior to 1.2.9. I could get away with flushing exactly one byte, then letting the next call to deflate/flush handle the rest of the data that deflatParams wanted to output. The documented interface to deflatParams in zlib 1.2.9 (and greater) has broken backward compatibility and need to be handled differently. I'm about half way through that change.
From: paul [...] city-fan.org
2.072 is passing all my test builds now, with zlib back to 1.2.3. Thanks.
On Mon Feb 13 04:58:32 2017, paul@city-fan.org wrote: Show quoted text
> 2.072 is passing all my test builds now, with zlib back to 1.2.3. Thanks.
Thanks for that - my testing uses all zlib version from 1.2.0 through to 1.2.11 Paul