Skip Menu |

This queue is for tickets about the DBD-Pg CPAN distribution.

Report information
The Basics
Id: 44225
Status: resolved
Priority: 0/
Queue: DBD-Pg

People
Owner: greg [...] turnstep.com
Requestors: szbalint [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: 2.11.8
Fixed in: 2.12.0



Subject: array retrieval leaks memory
Hi, DBD::Pg leaks memory when it's accessing PostgreSQL array values. Creating the included table and running leak.t produces a constant and sizeable memory leak for me. When the placeholder is changed from 'foo' to 'bar', so that for the given row the array is NULL, the leak stops and the program has constant memory usage over time. I've used 5.8.8 and the latest DBD::Pg version: $ perl5.8.8 -MDBD::Pg -le 'print $DBD::Pg::VERSION' 2.11.8
Subject: test_leak.sql
drop table if exists test_leak cascade; create table test_leak ( id serial NOT NULL, numbers integer[], string text NOT NULL, CONSTRAINT test_leak_pkey PRIMARY KEY (id) ); INSERT INTO test_leak (string,numbers) VALUES ('foo', '{}'); INSERT INTO test_leak (string) VALUES ('bar');
Subject: leak.t
use strict; use warnings; use DBI; my $dbh = DBI->connect('dbi:Pg:host=localhost;dbname=test;port=5432;',"postgres","postgres") or die($DBI::errstr); my $count = 0; my $sth = $dbh->prepare("SELECT * FROM test_leak WHERE string = ?"); while (1) { { $sth->execute('foo'); my $res = $sth->fetchall_arrayref({}); if ($res && @{$res}) { $count += @{$res}; } else { exit; } print "$count\n" if ($count % 1000 == 0); } }
Thanks for the report (and test case!). I think this is fixed by making pg_destringify_array return a non-incremented version of the arrayref. This is committed in r12613. Please confirm that it fixes the problem you are seeing and does not cause any others. It's simple enough to patch if you don't want to download the whole of HEAD: just change the last line of pg_destringify_array from return newRV((SV*)av) to return newRV_noinc((SV*)av);
On Wed Mar 18 15:28:13 2009, greg@turnstep.com wrote: Show quoted text
> Thanks for the report (and test case!). I think this is fixed by making > pg_destringify_array return a non-incremented version of the arrayref. > This is committed in r12613. Please confirm that it fixes the problem > you are seeing and does not cause any others. It's simple enough to > patch if you don't want to download the whole of HEAD: just change the > last line of pg_destringify_array from return newRV((SV*)av) to return > newRV_noinc((SV*)av);
It seems that this fixes the problem indeed. Cheers.
On Fri Mar 20 01:41:00 2009, SZBALINT wrote: Show quoted text
> It seems that this fixes the problem indeed.
Actually, scratch that. I was accidentally testing with the row that had a NULL array, not the one that contained the data. The noinc fix doesn't seem to fix the problem unfortunately, something else is causing it.
Show quoted text
> The noinc fix doesn't seem to fix the problem unfortunately, something > else is causing it.
That's unfortunate. It fixed it on my system, so I guess at this point we need a better test case?
On Wed Mar 25 11:12:18 2009, greg@turnstep.com wrote: Show quoted text
> > The noinc fix doesn't seem to fix the problem unfortunately, something > > else is causing it.
> > That's unfortunate. It fixed it on my system, so I guess at this point > we need a better test case?
You're correct. Testing is hard :) This is what I think happens at the moment: The noinc change appears to fix the problem that is presented by the testcase, which is a non-NULL postgres array value, that happens to contain no elements. ('{}'). I still see a memory leak if the array does contain one or more elements though. I've attached a test case where the array contains elements.
drop table if exists test_leak cascade; create table test_leak ( id serial NOT NULL, numbers integer[], string text NOT NULL, CONSTRAINT test_leak_pkey PRIMARY KEY (id) ); INSERT INTO test_leak (string,numbers) VALUES ('foo', '{}'); INSERT INTO test_leak (string) VALUES ('bar'); INSERT INTO test_leak (string,numbers) VALUES ('baz', '{1}');
use strict; use warnings; use DBI; my $dbh = DBI->connect('dbi:Pg:host=localhost;dbname=test;port=5432;',"postgres","postgres") or die($DBI::errstr); my $count = 0; my $sth = $dbh->prepare("SELECT * FROM test_leak WHERE string = ?"); while (1) { { $sth->execute('baz'); my $res = $sth->fetchall_arrayref({}); if ($res && @{$res}) { $count += @{$res}; } else { exit; } print "$count\n" if ($count % 1000 == 0); } }
Please see if the changes in r12635 have fixed the problem. I mortalized the temporary strings we use to populate the sub-parts of the array: http://git.postgresql.org/?p=dbdpg.git;a=commitdiff;h=3508ab706546d898a217ab95815a76d56129541a
On Thu Mar 26 12:29:53 2009, greg@turnstep.com wrote: Show quoted text
> Please see if the changes in r12635 have fixed the problem. I > mortalized > the temporary strings we use to populate the sub-parts of the array: > >
http://git.postgresql.org/?p=dbdpg.git;a=commitdiff;h=3508ab706546d898a217ab95815a76d56129541a Looks like this commit indeed fixes the memory leak shown by the updated test case. Thank you for the fix and the quick response time!