Skip Menu |

This queue is for tickets about the Parse-RecDescent CPAN distribution.

Report information
The Basics
Id: 53710
Status: resolved
Priority: 0/
Queue: Parse-RecDescent
People
Owner: Nobody in particular
Requestors: agruber [...] tbi.univie.ac.at
Cc:
AdminCc:
Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)

History Show all quoted text
  Thu Jan 14 22:03:32 2010 agruber [...] tbi.univie.ac.at - Ticket created
Subject: Severe Memory leak
Date: Fri, 15 Jan 2010 04:03:05 +0100
To: bug-Parse-RecDescent [...] rt.cpan.org
From: Andreas Gruber <agruber [...] tbi.univie.ac.at>
Below is a simple linux perl program that shows that this module has some memory leaks: use warnings; use strict; use Parse::RecDescent; for my $i ( 1 .. 1000 ) { my $grammar = q { # GRAMMAR SPECIFICATION HERE }; my $parser = new Parse::RecDescent ($grammar); my $mem = `ps h -o size $$`; chomp($mem); $mem = sprintf( "$i :: %.1f", $mem / 1024 ); print "$mem\n"; } Even on an empty grammar (just comment): Iteration 1: 1.8 MB Iteration 1000: 5.1 MB
  Thu Jan 14 23:05:39 2010 damian [...] conway.org - Correspondence added
Subject: Re: [rt.cpan.org #53710] Severe Memory leak
Date: Fri, 15 Jan 2010 15:04:53 +1100
To: bug-Parse-RecDescent [...] rt.cpan.org
From: Damian Conway <damian [...] conway.org>
Show quoted text
> Below is a simple linux perl program that shows that this module has > some memory leaks:
This is not a memory leak, per se. Every time you create a new grammar, the methods implementing the rules of that grammar are installed in their own unique namespace. Because these methods are installed in a global namespace, they are never removed, and the memory increase you are seeing is (partly) a result of that. I have tried instrumenting the object's destructor to clean up those namespaces. This eliminates about half the leakage on my system, but I cannot find where the other leak is occurring. I will be happy to patch it if you (or anyone else) can locate the remaining source of memory use. Meanwhile, please try the attached alpha of RecDescent.pm and see if it decreases leakage on your system too. Thanks for the report, Damian

Message body is not shown because sender requested not to inline it.

  Thu Jan 14 23:05:50 2010 The RT System itself - Status changed from 'new' to 'open'
  Fri Feb 19 08:29:11 2010 michael.rigoni+bitcard [...] mancalanetworks.com - Correspondence added
From: michael.rigoni+bitcard [...] mancalanetworks.com
Hi, I had the same memory leak problem. I use a class of mine that inherits from Parse::RecDescent. I added the following DESTROY method to clean things up: ------------------------------------------------- package myParser; use base qw(Parse::RecDescent); use Class::Unload; .... .... sub DESTROY { my $self = shift; Class::Unload->unload( $self->{namespace} ) } ------------------------------------------------- This seems to work for me, however I only use one instance of a particular grammar at a time, so I can safely unload the Namespace when that instance is no longer used. To make things clean, one should check that no object uses that class, but I'm not quite sure how to perform that... Hope this helps. Michael
  Fri Feb 19 15:21:30 2010 damian [...] conway.org - Correspondence added
Subject: Re: [rt.cpan.org #53710] Severe Memory leak
Date: Sat, 20 Feb 2010 07:20:33 +1100
To: bug-Parse-RecDescent [...] rt.cpan.org
From: Damian Conway <damian [...] conway.org>
Show quoted text
> I use a class of mine that inherits from Parse::RecDescent. I added the > following DESTROY method to clean things up:
Thanks for the useful feedback , Michael. You might like to try the latest release of P::RD (uploaded to CPAN a few days ago). It has, I believe, a destructor that handles this issue correctly. All the best, Damian
  Mon Apr 18 05:14:48 2011 JZOBEL [...] cpan.org - Correspondence added
Am Fr 19. Feb 2010, 15:21:30, damian@conway.org schrieb: Show quoted text
> It has, I > believe, a destructor that handles this issue correctly.
Unfortunately this seems to be wrong. At least my module (Cindy 0.15) is still leaking memory with 1.965001. I will however simply work around this by precompiling, which looks like a good idea anyway. Sincerely, Joachim
  Wed Feb 08 23:57:57 2012 jtbraun [...] CPAN.org - Correspondence added
It would appear that at least a portion of the leaked memory can be directly attributed to the fact that the Parse::RecDescent:namespace%d packages implement an AUTOLOAD method. If I rename this method in the generated code (temporarily) to AUTOLOAD2, I no longer see the ever-increasing memory (Perl v5.14.2). Even with a simple grammar defined, this one change keeps the memory usage in the provided test constant. It's unclear if this is a Perl bug, or a necessary feature/sideeffect of defining AUTOLOAD. To be determined.
  Thu Feb 09 00:14:59 2012 jtbraun [...] CPAN.org - Correspondence added
On closer inspection, it seems to be dependent on code like the following being eval'ed for different $package values substituted in. Something do do with the fact that the special variable $AUTOLOAD is used by the special sub AUTOLOAD. Renaming either one prevents an increase in memory usage. Leaving both named AUTOLOAD results in an apparent leak. qq{ package $package; use vars qw(\$AUTOLOAD ); { \*${package}::AUTOLOAD = sub { goto \&{\$AUTOLOAD}; }; } On Wed Feb 08 23:57:57 2012, jtbraun wrote: Show quoted text
> It would appear that at least a portion of the leaked memory can be > directly attributed to the fact that the Parse::RecDescent:namespace%d > packages implement an AUTOLOAD method. If I rename this method in the > generated code (temporarily) to AUTOLOAD2, I no longer see the > ever-increasing memory (Perl v5.14.2). Even with a simple grammar > defined, this one change keeps the memory usage in the provided test > constant. > > It's unclear if this is a Perl bug, or a necessary feature/sideeffect of > defining AUTOLOAD. To be determined.
  Thu Feb 09 00:54:03 2012 jtbraun [...] CPAN.org - Correspondence added
I've filed the following bug report (with simplified test case) against perl5: https://rt.perl.org/rt3/Public/Bug/Display.html?id=110248 We'll see what p5p and crew have to say.
  Sun Feb 19 15:04:33 2012 jtbraun [...] CPAN.org - Correspondence added
perl5-porters identified the issue with $AUTOLOAD. Using $AUTOLOAD inside a method named AUTOLOAD has an optimization that stores a reference to the AUTOLOAD glob in the compiled AUTOLOAD sub. The result is a circular reference for the AUTOLOAD glob in each namespace. https://rt.perl.org/rt3/Public/Bug/Display.html?id=110248 Using this workaround will potentially cause a small performance hit, as the AUTOLOAD dispatch now has to look up the AUTOLOAD{SCALAR} glob entry. I'll look into whether there is a significant difference in performance, and either provide a compilation option to avoid this leak, or make it the default behavior, depending on how much performance is affected. There still appears to be other (smaller) leaks, which I'll continue to investigate.
  Sun Feb 19 16:18:57 2012 jtbraun [...] CPAN.org - Correspondence added
FYI: the fix for the AUTOLOAD circular reference. https://github.com/jtbraun/Parse-RecDescent/commit/589ac9b6d48f8028d5a52fbb9cc15353316bd7c8 This did cause about a 1% performance hit in user time for frequenct calls into a simple example parser via $parser->start_rule. The bug remains open, as there appear to be other, slower leaks as mentioned previously.
  Tue Feb 21 00:48:46 2012 jtbraun [...] CPAN.org - Correspondence added
There appears to be another circular reference leak in Perl itself, documented here: https://rt.perl.org/rt3/Ticket/Display.html?id=92708 I've found what appears to be a workaround for the issue, documented here: https://rt.perl.org/rt3/Ticket/Display.html?id=111206 Changes to Parse::RecDescent: https://github.com/jtbraun/Parse-RecDescent/commit/682b23b4f76542172ee534ea9ba83d4e40468053 https://github.com/jtbraun/Parse-RecDescent/commit/077db065a4b680f48dbbb5b7ddf4bdbbf47e1a9a
  Tue Feb 21 00:55:13 2012 jtbraun [...] CPAN.org - Correspondence added
The recent $AUTOLOAD and @ISA leak workarounds seem to have stabilized the provided test case and some simple grammars, but more leaks appear to exist. This issue will remain open for the time being.
  Tue Feb 21 00:55:14 2012 jtbraun [...] CPAN.org - Status changed from 'open' to 'resolved'
  Tue Feb 21 00:55:38 2012 jtbraun [...] CPAN.org - Status changed from 'resolved' to 'open'
  Tue Feb 21 01:54:27 2012 jtbraun [...] CPAN.org - Correspondence added
I tried some more complicated grammars, and found another circular reference between subrules. Fix here: https://github.com/jtbraun/Parse-RecDescent/commit/c6cebc06d848ea9433c4e8489ed8507dee0b0009 These changes will all appear in the next version of Parse::RecDescent. I believe that this fixes many of the common causes of memory leaks within Parse::RecDescent, so I'm resolving this issue. If you find more, please let me know. Thank you for the bug report, and for including a test case. Jeremy
  Tue Feb 21 01:54:28 2012 jtbraun [...] CPAN.org - Status changed from 'open' to 'resolved'