Bug #72595 for WWW-Scripter-Plugin-JavaScript: WWW::Scripter with Javascript plugin "leaks" memory

Mon Nov 21 18:14:21 2011 stas [...] stason.org - Ticket created

Subject:	WWW::Scripter with Javascript plugin "leaks" memory
Date:	Mon, 21 Nov 2011 15:13:48 -0800
To:	bug-WWW-Scripter-Plugin-JavaScript [...] rt.cpan.org
From:	Stas Bekman <stas [...] stason.org>

Hi Father, It looks like JS engine doesn't free up the memory it used, e.g. a simple script below adds 30-50MB each run, even though no history is saved (at least according to the docs). Memory leaking has been observed in top(1). #!/usr/bin/perl -- use strict; use warnings; use WWW::Mechanize; use WWW::Scripter; my $mech = WWW::Scripter->new(max_docs => 1, max_history => 1); $mech->use_plugin('JavaScript'); warn "engine started\n"; for (1..10) { warn "run $_\n"; $mech->get('http://yahoo.com'); $mech->clear_history; } turning JS off fixes that, but defeats the purpose ;) I'm using all the latest versions: WWW-Scripter-Plugin-JavaScript-0.007 WWW-Scripter-0.024 This is perl 5, version 12, subversion 4 (v5.12.4) built for i686-linux-gnu-thread-multi-64int (with 45 registered patches, see perl -V for more detail) Linux 3.0.0-12-generic-pae #20-Ubuntu SMP Fri Oct 7 16:37:17 UTC 2011 i686 i686 i386 GNU/Linux Thanks -- Show quoted text

________________________________________________ Stas Bekman http://stasosphere.com http://stason.org http://chestofbooks.com http://vitalitylink.com http://healingcloud.com

Sun Nov 27 17:16:24 2011 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Mon Nov 21 18:14:21 2011, STAS wrote: Show quoted text

> Hi Father, > > It looks like JS engine doesn't free up the memory it used, e.g. a > simple script below adds 30-50MB each run, even though no history is > saved (at least according to the docs). Memory leaking has been observed > in top(1). > > #!/usr/bin/perl -- > > use strict; > use warnings; > > use WWW::Mechanize; > use WWW::Scripter; > > my $mech = WWW::Scripter->new(max_docs => 1, max_history => 1); > $mech->use_plugin('JavaScript'); > > warn "engine started\n"; > > for (1..10) { > warn "run $_\n"; > $mech->get('http://yahoo.com'); > $mech->clear_history; > } > > turning JS off fixes that, but defeats the purpose ;) > > I'm using all the latest versions: > > WWW-Scripter-Plugin-JavaScript-0.007 > WWW-Scripter-0.024

The problem is that JE creates circularities. I tried doing a simple brute-force destruction, but caching all JE objects in a simple hash and then undefining the contents of them all afterwards, but things are still leaking. It’s going to take me some time to track down the remaining leaks.

Sun Nov 27 17:16:25 2011 The RT System itself - Status changed from 'new' to 'open'

Sun Nov 27 17:27:42 2011 stas [...] stason.org - Correspondence added

Subject:	Re: [rt.cpan.org #72595] WWW::Scripter with Javascript plugin "leaks" memory
Date:	Sun, 27 Nov 2011 14:27:07 -0800
To:	bug-WWW-Scripter-Plugin-JavaScript [...] rt.cpan.org
From:	Stas Bekman <stas [...] stason.org>

On 11-11-27 02:16 PM, Father Chrysostomos via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=72595> > > On Mon Nov 21 18:14:21 2011, STAS wrote:

>> Hi Father, >> >> It looks like JS engine doesn't free up the memory it used, e.g. a >> simple script below adds 30-50MB each run, even though no history is >> saved (at least according to the docs). Memory leaking has been observed >> in top(1).

[...] Show quoted text

> The problem is that JE creates circularities. I tried doing a simple brute-force destruction, > but caching all JE objects in a simple hash and then undefining the contents of them all > afterwards, but things are still leaking. It’s going to take me some time to track down the > remaining leaks.

Yes, those are difficult to remove. Perhaps some hacks we used in the modperl code could be of help? The auto-generated ModPerl/Util.pm has this code: sub unload_package_pp { my $package = shift; no strict 'refs'; my $tab = \%{ $package . '::' }; # below we assign to a symbol first before undef'ing it, to avoid # nuking aliases. If we undef directly we may undef not only the # alias but the original function as well for (keys %$tab) { #Skip sub stashes next if /::$/; my $fullname = join '::', $package, $_; # code/hash/array/scalar might be imported make sure the gv # does not point elsewhere before undefing each if (%$fullname) { *{$fullname} = {}; undef %$fullname; } if (@$fullname) { *{$fullname} = []; undef @$fullname; } if ($$fullname) { my $tmp; # argh, no such thing as an anonymous scalar *{$fullname} = \$tmp; undef $$fullname; } if (defined &$fullname) { no warnings; local $^W = 0; if (defined(my $p = prototype $fullname)) { *{$fullname} = eval "sub ($p) {}"; } else { *{$fullname} = sub {}; } undef &$fullname; } if (*{$fullname}{IO}) { if (fileno $fullname) { close $fullname; } } } #Wipe from %INC $package =~ s[::][/]g; $package .= '.pm'; delete $INC{$package}; } You can see it being used in http://cpansearch.perl.org/src/PHRED/mod_perl-2.0.5/ModPerl-Registry/lib/ModPerl/RegistryCooker.pm sub flush_namespace_normal { my $self = shift; $self->debug("flushing namespace") if DEBUG & D_NOISE; ModPerl::Util::unload_package($self->{PACKAGE}); } perhaps this could help. I haven't tried applying it for your module. -- Show quoted text

________________________________________________ Stas Bekman http://stasosphere.com http://stason.org http://chestofbooks.com http://vitalitylink.com http://healingcloud.com

Mon Dec 05 01:56:40 2011 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Sun Nov 27 17:27:42 2011, STAS wrote: Show quoted text

> On 11-11-27 02:16 PM, Father Chrysostomos via RT wrote:

> > <URL: https://rt.cpan.org/Ticket/Display.html?id=72595> > > > > On Mon Nov 21 18:14:21 2011, STAS wrote:

> >> Hi Father, > >> > >> It looks like JS engine doesn't free up the memory it used, e.g. a > >> simple script below adds 30-50MB each run, even though no history

> is

> >> saved (at least according to the docs). Memory leaking has been

> observed

> >> in top(1).

> [...]

> > The problem is that JE creates circularities. I tried doing a

> simple brute-force destruction,

> > but caching all JE objects in a simple hash and then undefining the

> contents of them all

> > afterwards, but things are still leaking. It’s going to take me

> some time to track down the

> > remaining leaks.

> > Yes, those are difficult to remove.

... Show quoted text

> perhaps this could help. I haven't tried applying it for your module.

Thank you, but it looks as though that code is for unloading modules. In my case it’s a matter of freeing a twisted maze of circular references. I think I’ve managed to do it. Try the new JE 0.056 and WWW::Scripter::Plugin::JavaScript 0.008. The leaks from JE I believe are completely gone, but I think something is still leaking slightly as memory usage goes up 1MB for me for each page. In fixing JE I actually discovered a perl bug. :-)

Mon Dec 05 01:56:41 2011 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Status changed from 'open' to 'resolved'

Mon Dec 05 20:29:45 2011 stas [...] stason.org - Correspondence added

Subject:	Re: [rt.cpan.org #72595] WWW::Scripter with Javascript plugin "leaks" memory
Date:	Mon, 05 Dec 2011 17:29:33 -0800
To:	bug-WWW-Scripter-Plugin-JavaScript [...] rt.cpan.org
From:	Stas Bekman <stas [...] stason.org>

On 11-12-04 10:56 PM, Father Chrysostomos via RT wrote: [...] Show quoted text

> In my case it’s a matter of freeing a twisted maze of circular references. > > I think I’ve managed to do it. Try the new JE 0.056 and WWW::Scripter::Plugin::JavaScript > 0.008. > > The leaks from JE I believe are completely gone, but I think something is still leaking slightly > as memory usage goes up 1MB for me for each page.

It's much better, but I still get 2-3MB/page on that test url of http://yahoo.com - 25 runs on it gave me about 60MB in leaks (started to measure after the first run was complete). On another site it was even bigger. So I still can't use it as it'd leak too much memory pretty fast. Show quoted text

> In fixing JE I actually discovered a perl bug. :-)

great! -- Show quoted text

________________________________________________ Stas Bekman http://stasosphere.com http://stason.org http://chestofbooks.com http://vitalitylink.com http://healingcloud.com

Mon Dec 05 20:29:46 2011 The RT System itself - Status changed from 'resolved' to 'open'

Sun Dec 11 21:30:10 2011 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Mon Dec 05 20:29:45 2011, STAS wrote: Show quoted text

> On 11-12-04 10:56 PM, Father Chrysostomos via RT wrote: > [...]

> > In my case it’s a matter of freeing a twisted maze of circular

> references.

> > > > I think I’ve managed to do it. Try the new JE 0.056 and

> WWW::Scripter::Plugin::JavaScript

> > 0.008. > > > > The leaks from JE I believe are completely gone, but I think

> something is still leaking slightly

> > as memory usage goes up 1MB for me for each page.

> > It's much better, but I still get 2-3MB/page on that test url of > http://yahoo.com - 25 runs on it gave me about 60MB in leaks (started > to > measure after the first run was complete). On another site it was even > bigger. So I still can't use it as it'd leak too much memory pretty > fast.

Well, I had another stab at it. (You don’t want to know how much time I spent.) I managed to track down a few leaks in JE that Devel::Cycle helped me find, but there are still a few left that it cannot see, probably due to field hashes. Devel::Leak::Object produces a long list at the end of the script. I haven’t bothered making a release yet, since my local fixes don’t make enough of a difference. I’ve just discovered Devel::FindRef. Maybe I can try using that when I get a chance. Show quoted text

>

> > In fixing JE I actually discovered a perl bug. :-)

> > great! >

Sun Dec 11 21:31:06 2011 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Mon Dec 05 20:29:45 2011, STAS wrote: Show quoted text

> On 11-12-04 10:56 PM, Father Chrysostomos via RT wrote:

> > In fixing JE I actually discovered a perl bug. :-)

> > great! >

And that’s #105918, by the way.

Sun Jan 01 16:01:21 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Sun Dec 11 21:30:10 2011, SPROUT wrote: Show quoted text

> On Mon Dec 05 20:29:45 2011, STAS wrote:

> > On 11-12-04 10:56 PM, Father Chrysostomos via RT wrote: > > [...]

> > > In my case it’s a matter of freeing a twisted maze of circular

> > references.

> > > > > > I think I’ve managed to do it. Try the new JE 0.056 and

> > WWW::Scripter::Plugin::JavaScript

> > > 0.008. > > > > > > The leaks from JE I believe are completely gone, but I think

> > something is still leaking slightly

> > > as memory usage goes up 1MB for me for each page.

> > > > It's much better, but I still get 2-3MB/page on that test url of > > http://yahoo.com - 25 runs on it gave me about 60MB in leaks

> (started

> > to > > measure after the first run was complete). On another site it was

> even

> > bigger. So I still can't use it as it'd leak too much memory pretty > > fast.

> > Well, I had another stab at it. (You don’t want to know how much time > I spent.) I managed to > track down a few leaks in JE that Devel::Cycle helped me find, but > there are still a few left that > it cannot see, probably due to field hashes. Devel::Leak::Object > produces a long list at the > end of the script. > > I haven’t bothered making a release yet, since my local fixes don’t > make enough of a > difference. > > I’ve just discovered Devel::FindRef. Maybe I can try using that when > I get a chance.

I finally managed to track down the remaining memory leak. It was in WWW::Scripter::Frames, and the reference loop included two field hashes, which is why it was so hard to track down. Could you try JE 0.057 and WWW::Scripter 0.025 and see whether they work for you?

Sun Jan 01 16:01:22 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Status changed from 'open' to 'resolved'

Sun Jan 01 23:48:26 2012 stas [...] stason.org - Correspondence added

Subject:	Re: [rt.cpan.org #72595] WWW::Scripter with Javascript plugin "leaks" memory
Date:	Sun, 01 Jan 2012 20:48:09 -0800
To:	bug-WWW-Scripter-Plugin-JavaScript [...] rt.cpan.org
From:	Stas Bekman <stas [...] stason.org>

Message body is not shown because it is too large.

Sun Jan 01 23:48:27 2012 The RT System itself - Status changed from 'resolved' to 'open'

Mon Jan 02 01:24:01 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Sun Jan 01 23:48:26 2012, STAS wrote: Show quoted text

>

> > I finally managed to track down the remaining memory leak. It was

> in WWW::Scripter::Frames,

> > and the reference loop included two field hashes, which is why it

> was so hard to track down.

> > > > Could you try JE 0.057 and WWW::Scripter 0.025 and see whether they

> work for you? > > Awesome - no more leaking - good job, Father! > > Except after installing the latest versions of the above, running the > same script i posted in the bug report, I get a bunch of warnings > printed all the time:

Oops. Sorry. I left some debugging code behind by mistake. I removed it in JE 0.058.

Mon Jan 02 01:24:02 2012 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Status changed from 'open' to 'resolved'