Skip Menu |

This queue is for tickets about the perl-ldap CPAN distribution.

Report information
The Basics
Id: 84410
Status: resolved
Priority: 0/
Queue: perl-ldap

People
Owner: Nobody in particular
Requestors: jeffrey.a.dunham [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: 0.55



Subject: PersistentSearch Memory Issue
Date: Wed, 3 Apr 2013 09:09:45 -0700
To: bug-perl-ldap [...] rt.cpan.org
From: Jeffrey Dunham <jeffrey.a.dunham [...] gmail.com>
I use Persistent search to query our change log and do some actions based off of that. Checking the heap of my process it gets large, growing pretty quick ( LDAP is changed extremely often here so our changelog is large ) after 2 days for example I am using 8 GB of memory and eventually my server starts swapping From what I can tell it appears persistentsearch is the culprit, I am using perl 5.8 and Net::LDAP 0.53. My code looks like: $persist = Net::LDAP::Control::PersistentSearch->new(changeTypes => 1, changesOnly => 0, returnECs => 1 ); $sort = Net::LDAP::Control::Sort->new(order => "changeNumber"); if (defined $last_cn) { my $next_cn = last_cn + 1; $mesg = $ldapconn->search( base => "cn=changelog", filter => "(&(objectClass=changelogentry)(changeNumber>=$next_cn))", callback => \&process_entry, control => [ $persist , $sort ] ); die "error: ",$mesg->code(),": ",$mesg->error() if ($mesg->code()); } This process simply eats memory. This is a child process that does this, so what we've ended up doing is after a set amount of changes, we restart the child, which does work, but it's a hacky solution.
Subject: Re: [rt.cpan.org #84410] PersistentSearch Memory Issue
Date: Wed, 3 Apr 2013 11:23:37 -0500
To: bug-perl-ldap [...] rt.cpan.org
From: Graham Barr <gbarr [...] pobox.com>
Show quoted text
> > $mesg = $ldapconn->search( > base => "cn=changelog", > filter => > "(&(objectClass=changelogentry)(changeNumber>=$next_cn))", > callback => \&process_entry, > control => [ $persist , $sort ] ); > die "error: ",$mesg->code(),": ",$mesg->error() if ($mesg->code()); > } > > This process simply eats memory. This is a child process that does this, > so what we've ended up doing is after a set amount of changes, we restart > the child, which does work, but it's a hacky solution.
You do not show your definition of process_entry, but unless it is removing the entries from the search object, they will consume memory. See http://search.cpan.org/~marschap/perl-ldap-0.54/lib/Net/LDAP/FAQ.pod#USING_THE_CALLBACK_SUBROUTINE_APPROACH Which shows an example of a callback, which ends with $mesg->pop_entry; and then states As you can see the example is straightforward and it does not waste time waiting for all of the entries to be returned. However if the pop_entry method is not used the callback approach can allocate a lot of memory to the search request Graham.
Subject: Re: [rt.cpan.org #84410] PersistentSearch Memory Issue
Date: Thu, 4 Apr 2013 10:56:36 -0700
To: bug-perl-ldap [...] rt.cpan.org
From: Jeffrey Dunham <jeffrey.a.dunham [...] gmail.com>
Thank you so much that did fix our issue. Is it possible to have that in the PersistentSearch example itself? I can't imagine why someone would want to have a callback on a persistent search where you didn't get rid of an entry after you have processed it. Again thank you. -Jeff On Wed, Apr 3, 2013 at 9:23 AM, Graham Barr via RT < bug-perl-ldap@rt.cpan.org> wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=84410 > >
> > > > $mesg = $ldapconn->search( > > base => "cn=changelog", > > filter => > > "(&(objectClass=changelogentry)(changeNumber>=$next_cn))", > > callback => \&process_entry, > > control => [ $persist , $sort ] ); > > die "error: ",$mesg->code(),": ",$mesg->error() if ($mesg->code()); > > } > > > > This process simply eats memory. This is a child process that does this, > > so what we've ended up doing is after a set amount of changes, we restart > > the child, which does work, but it's a hacky solution.
> > You do not show your definition of process_entry, but unless it is > removing the entries from the search object, they will consume memory. > > See > http://search.cpan.org/~marschap/perl-ldap-0.54/lib/Net/LDAP/FAQ.pod#USING_THE_CALLBACK_SUBROUTINE_APPROACH > > Which shows an example of a callback, which ends with $mesg->pop_entry; > and then states > > As you can see the example is straightforward and it does not waste > time > waiting for all of the entries to be returned. However if the pop_entry > method is not used the callback approach can allocate a lot of memory > to the search request > > > Graham. > > >
Hi, On Thu Apr 04 13:56:54 2013, jeffrey.a.dunham@gmail.com wrote: Show quoted text
> Thank you so much that did fix our issue. Is it possible to have that > in > the PersistentSearch example itself? I can't imagine why someone > would > want to have a callback on a persistent search where you didn't get > rid of > an entry after you have processed it. Again thank you.
thanks for reporting the bug. The example in the PersistentSearch.pm file will be updated in release 0.55 which I wil lrelease today. Best PEter