Skip Menu |

This queue is for tickets about the IO-Compress CPAN distribution.

Report information
The Basics
Id: 72945
Status: resolved
Priority: 0/
Queue: IO-Compress

People
Owner: Nobody in particular
Requestors: cpan [...] robm.fastmail.fm
Cc:
AdminCc:

Bug Information
Severity: Normal
Broken in: 2.020
Fixed in: (no value)



Subject: Compress::Zlib::memGunzip holds reference to input parameter
This code surprisingly leaks an object reference. sub a { my $b = shift; my $u = Compress::Zlib::memGunzip($b); $b = SomeObj->new(); return $u; } Because Compress::Zlib::memGunzip takes a reference to $_[0] and stores it somewhere in a global. That means there's a global ref held to $b, and when $b holds an object, that ref is held until the process exits or the next call to memGunzip which overwrites the global. This took ages to track down an annoying bug in my apache application because processes are long lived.
Hi, thanks for the report. You mention that "Compress::Zlib::memGunzip takes a reference to $_[0] and stores it somewhere in a global". Can't say I'm aware of any global that stores a reference to $_[0] in my code. Did you confirm that was happening in your investigation, or are you assuming that must be the case if you are seeing a leak? I'm having difficulty reproducing the memory leak you are getting. I ran the code below for 30 minutes and didn't see any memory growth in the process. Does the code below leak for you? Finally, what version of Perl are you running? What platform (linux/Windows etc). Are you running mod_perl? If so, what version? use Compress::Zlib qw(memGunzip $gzerrno) ; use IO::Compress::Gzip qw(:all); use IO::File ; sub a { my $b = shift; my $u = Compress::Zlib::memGunzip($b); $b = IO::File->new(">/tmp/x"); return $u; } my $gz ; gzip \"hello world\n" => \$gz or die "cannot compress\n";; while(1) { my $d = a($gz); }
Show quoted text
> You mention that "Compress::Zlib::memGunzip takes a reference to $_[0] > and stores it somewhere in a global". Can't say I'm aware of any global > that stores a reference to $_[0] in my code. Did you confirm that was > happening in your investigation, or are you assuming that must be the > case if you are seeing a leak?
Yes, I confirmed it. See example code below. Show quoted text
> I'm having difficulty reproducing the memory leak you are getting. I ran > the code below for 30 minutes and didn't see any memory growth in the > process. Does the code below leak for you?
So it's not a "leak" in that it doesn't grow forever. It's just that somewhere there is a single reference held to the *last* input to memGunzip. Normally that's not a problem, but in long lived processes (eg apache mod_perl handlers) it can be because it can mean an object lives longer than expected. Show quoted text
> Finally, what version of Perl are you running? What platform > (linux/Windows etc). Are you running mod_perl? If so, what version?
It's debian linux, perl 5.10.1. ----- use Compress::Zlib; # new object my $f = test->new(); # dummy so we can call memGunzip my $z = Compress::Zlib::memGzip(""); uz($z, $f); # undef object reference, should cause test::DESTROY to be called # straight away because there should be no references left. # but that's not what happens, object is only destroyed in # global destruction phase warn "about to undef last reference"; $f = undef; warn "after undefing last reference"; sub uz { # $_[0] == gzip buffer, $_[1] == object reference my ($zp, $rp) = @_; # unzip the buffer, don't care about results. # the bug is that this ends up storing a reference to $zp somewhere # in a global Compress::Zlib::memGunzip($zp); # replace $zp with an object reference. because memGunzip holds a # reference to $zip somewhere, we now actually have a reference to # the "test" object held somewhere! $zp = $rp; return; } # Test object that logs creation/destruction package test; sub new { my $self = bless {}, 'test'; warn "new $self"; return $self; } sub DESTROY { my $self = shift; warn "destroy $self"; } ----- What you see: $ perl /tmp/t.pl new test=HASH(0x2059f70) at /tmp/t.pl line 41. about to undef last reference at /tmp/t.pl line 15. after undefing last reference at /tmp/t.pl line 17. destroy test=HASH(0x2059f70) at /tmp/t.pl line 47 during global destruction. Note that the object is only destroyed "during global destruction", not when the actual last reference is undef'd. Now if you comment out the call to memGunzip you see: $ perl /tmp/t.pl new test=HASH(0x1c25f70) at /tmp/t.pl line 41. about to undef last reference at /tmp/t.pl line 15. destroy test=HASH(0x1c25f70) at /tmp/t.pl line 47. after undefing last reference at /tmp/t.pl line 17. Which is the correct expected behaviour, the object is destroyed when the last reference to it is undef'd.
Brilliant test case, many thanks. I can now reproduce the issue on 5.10.1. Interestingly all is fine with perl 5.14.0 or better. A quick test with my collection of Perl binaries shows it starts working from 5.13.4. I added a Devel::Peek::Dump call directly before and after this line in your test script $zp = $rp; This is the before dump of $zp SV = PV(0x9e26b80) at 0x9e549d8 REFCNT = 3 FLAGS = (PADMY,POK,OOK,pPOK) OFFSET = 18 PV = 0x9ee0162 ( "\37\213\10\0\0\0\0\0\0\n\0\0\0\0\0\0\0\22" . ) ""\0 CUR = 0 LEN = 6 and this is the after SV = PV(0x9e26b80) at 0x9e549d8 REFCNT = 3 FLAGS = (PADMY,ROK) RV = 0x9d37270 SV = PVHV(0x9d3ef98) at 0x9d37270 REFCNT = 3 FLAGS = (OBJECT,SHAREKEYS) STASH = 0x9f69000 "test" ARRAY = 0x0 KEYS = 0 FILL = 0 MAX = 7 RITER = -1 EITER = 0x0 PV = 0x9d37270 "" CUR = 0 LEN = 0 Note the reference count set to 3 in the latter. That looks like a Perl bug to me. Compare that with the output from 5.13.4 SV = PV(0x9b87a98) at 0x9d4f348 REFCNT = 1 FLAGS = (PADMY,POK,OOK,pPOK) OFFSET = 18 PV = 0x9cbc3a2 ( "\37\213\10\0\0\0\0\0\0\n\0\0\0\0\0\0\0\22" . ) ""\0 CUR = 0 LEN = 6 SV = PV(0x9b87a98) at 0x9d4f348 REFCNT = 1 FLAGS = (PADMY,ROK) RV = 0x9b3a188 SV = PVHV(0x9b41eb0) at 0x9b3a188 REFCNT = 3 FLAGS = (OBJECT,SHAREKEYS) STASH = 0x9d58e18 "test" ARRAY = 0x0 KEYS = 0 FILL = 0 MAX = 7 RITER = -1 EITER = 0x0 PV = 0x9b3a188 "" CUR = 0 LEN = 0 Note the REFCNT set correctly se to 1 this time. If this is a Perl bug, as I supect, your best bet is to either upgrade to 5.14, or restructure your code to avoid making that assignment. Will see if I can run a git bisect to see what change was responsible for fixing the issue. Paul
On Tue Dec 06 17:40:42 2011, PMQS wrote: Show quoted text
> Note the reference count set to 3 in the latter. That looks like a
Perl bug to me. The thing is, that refcount is the number of references held to the $zp variable itself. So when you do $zp = $rp, it doesn't change the number of references to $zp at all, it's only changing the content of the $zp scalar (from a string to a reference) The more interesting thing is if you do a Dump before the memGunzip call and after the call. Here's what you see: Before: SV = PV(0x227c578) at 0x22a9d60 REFCNT = 1 FLAGS = (PADMY,POK,pPOK) PV = 0x1e992d0 "\37\213\10\0\0\0\0\0\0\377\3\0\0\0\0\0\0\0\0\0"\0 CUR = 20 LEN = 24 After: SV = PVIV(0x1e83928) at 0x22a9d60 REFCNT = 3 FLAGS = (PADMY,POK,OOK,pPOK) IV = 18 (OFFSET) PV = 0x1e992e2 ( "\37\213\10\0\0\0\0\0\0\377\0\0\0\0\0\0\0\0" . ) ""\0 CUR = 0 LEN = 6 So it's the memGunzip call that adds extra references to $zp (2 extra in fact!), which is what I was seeing happening. It's possible to simulate this sort of bug by doing something like: sub addevilref { our $someglobal = \$_[0]; return "bar"; } If you now do something like: sub { my $a = "foo"; addevilref($a); $a = SomeObj->new(); } Then SomeObj won't actually be destroyed at the end of the sub like you might expect, and this is what we see memGunzip doing. Now given that different versions of perl fix it, I can see one of two options: 1. It's an internal perl bug 2. It's a bug/change of behavior in some core module included with perl Compress::Zlib uses Hope you can track it down :)
It's a Perl core issue - this is the fix that makes the issue go away. commit 2154eca77956ce145743765bea9ce269e6227984 Author: Eric Brine <ikegami@adaelis.com> Date: Sat Jul 31 01:56:43 2010 -0700 Fix untimely destruction introduced by lvalue ops [RT#67838] by returning a TEMP instead of using TARG. Made appropriate TODO tests live. :100644 100644 c1a357cc3eaa9726832fbd1f58d102fa54b7707b 903144ca86c0546ae0f51e81c7cfdcf92ac1b3a1 M doop.c :100644 100644 2dfca4cbc24e588fef7b4a29eb7d73257a430cb0 0da8bba494d7b8b4d424a6d34dec124e657f384e M pp.c :040000 040000 297b82e827da6a96bf5e598e6717bd34fcc8c1ae 6752b3a91a5bd07f2628d0fa2acb8bac79c00ac2 M t
On Thu Dec 08 03:23:56 2011, PMQS wrote: Show quoted text
> It's a Perl core issue - this is the fix that makes the issue go away.
Interesting. I wonder what construct Compress::Zlib is using that tickles this bug, it would be nice to try and work around it if it's not hard to do. Anyway, for now I've made this patch to Cache::FastMmap to work around it. https://github.com/robmueller/cache-fastmmap/commit/e44829c2a8e0b72ef63d9811bbfdcf79a81324ac Thanks for investigating this so deeply!
On Thu Dec 08 22:38:35 2011, ROBM wrote: Show quoted text
> On Thu Dec 08 03:23:56 2011, PMQS wrote:
> > It's a Perl core issue - this is the fix that makes the issue go
> away. > > Interesting. I wonder what construct Compress::Zlib is using that > tickles this bug, it would be nice to try and work around it if it's > not > hard to do.
Not sure what it is, but it has been there for a very very long time. I see the problem with Perl 5.6.0 Show quoted text
> Anyway, for now I've made this patch to Cache::FastMmap to work around > it.
Glad to hear there is a workaround. I assume upgrading to 5.14 isn't an option? Paul