Skip Menu |

This queue is for tickets about the JavaScript CPAN distribution.

Report information
The Basics
Id: 35571
Status: open
Priority: 0/
Queue: JavaScript

People
Owner: Nobody in particular
Requestors: CRAKRJACK [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Critical
Broken in: 1.06
Fixed in: (no value)



Subject: JavaScript leaks memory like a sieve
When you run this script, even though both JavaScript and Perl have discarded the "ThisObjectWillLeak" objects, they get left around in the perl interpreter side (don't know about the javascript interpreter side). 10 of these sample objects isnt a big deal, but when you're dealing with millions of objects / references that you just want to have Go Away, it gets kind of scary...
Subject: leaktest.pl
#!perl use strict; use warnings; use Devel::Leak::Object qw(GLOBAL_bless); use JavaScript qw(:all); { package ThisObjectWillLeak; sub new { my($class, $num) = @_; return bless { n => $num }, $class; } sub whoami { my $self = shift; return "Leaky object #$self->{n}"; } } { package ThisObjectWillNotLeak; sub new { my($class, $num) = @_; return bless { n => $num }, $class; } sub whoami { my $self = shift; return "Sealed object #$self->{n}"; } } our $rt = JavaScript::Runtime->new; our $cx = JavaScript::Context->new($rt); sub dont_leak_an_object { my $sealed = ThisObjectWillNotLeak->new(shift); $cx->bind_object(sealed => $sealed); return; } sub leak_an_object { my $leaky = ThisObjectWillLeak->new(shift); return $leaky; } sub write { my $msg = shift; print "$msg\n"; } foreach my $i ('ThisObjectWillLeak', 'ThisObjectWillNotLeak') { $cx->bind_class( name => $i, constructor => sub {}, methods => { whoami => $i->can('whoami') }, flags => JS_CLASS_NO_INSTANCE, package => $i ); } $cx->bind_function(leak_an_object => \&leak_an_object); $cx->bind_function(dont_leak_an_object => \&dont_leak_an_object); $cx->bind_function(write => \&write); foreach my $i (1 .. 10) { $cx->eval("leaky = leak_an_object($i)"); $cx->eval("dont_leak_an_object($i)"); $cx->eval("write(leaky.whoami())"); $cx->eval("write(sealed.whoami())"); }
Even if you throw away your JavaScript::Runtime and JavaScript::Context objects, the perl side still leaks the objects you had from the old context. I have modified the leaktest script to show this. Here's the object tracking out we create 5 contexts, each of which create 10 leaky objects before been thrown away, and: Tracked objects by class: Config 1 JavaScript::Context 1 JavaScript::Runtime 1 ThisObjectWillLeak 50 ThisObjectWillNotLeak 5 :-(
#!perl use strict; use warnings; use Devel::Leak::Object qw(GLOBAL_bless); use JavaScript qw(:all); { package ThisObjectWillLeak; sub new { my($class, $num) = @_; return bless { n => $num }, $class; } sub whoami { my $self = shift; return "Leaky object #$self->{n}"; } } { package ThisObjectWillNotLeak; sub new { my($class, $num) = @_; return bless { n => $num }, $class; } sub whoami { my $self = shift; return "Sealed object #$self->{n}"; } } our $rt; our $cx; sub dont_leak_an_object { my $sealed = ThisObjectWillNotLeak->new(shift); $cx->bind_object(sealed => $sealed); return; } sub leak_an_object { my $leaky = ThisObjectWillLeak->new(shift); return $leaky; } sub write { my $msg = shift; print "$msg\n"; } foreach my $cxloop (1 .. 5) { print "Context #$cxloop\n"; $rt = JavaScript::Runtime->new; $cx = JavaScript::Context->new($rt); foreach my $i ('ThisObjectWillLeak', 'ThisObjectWillNotLeak') { $cx->bind_class( name => $i, constructor => sub {}, methods => { whoami => $i->can('whoami') }, flags => JS_CLASS_NO_INSTANCE, package => $i ); } $cx->bind_function(leak_an_object => \&leak_an_object); $cx->bind_function(dont_leak_an_object => \&dont_leak_an_object); $cx->bind_function(write => \&write); foreach my $i (1 .. 10) { $cx->eval("leaky = leak_an_object($i)"); $cx->eval("dont_leak_an_object($i)"); $cx->eval("write(leaky.whoami())"); $cx->eval("write(sealed.whoami())"); } }
CC: "James A. Duncan" <jamesaduncan [...] mac.com>
Subject: Re: [rt.cpan.org #35571] JavaScript leaks memory like a sieve
Date: Fri, 2 May 2008 12:45:11 +0200
To: bug-JavaScript [...] rt.cpan.org
From: Claes Jakobsson <claes [...] versed.se>
On 1 maj 2008, at 20.23, CRAKRJACK via RT wrote: Show quoted text
> > Queue: JavaScript > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=35571 > > > Even if you throw away your JavaScript::Runtime and > JavaScript::Context > objects, the perl side still leaks the objects you had from the old > context. I have modified the leaktest script to show this. > > Here's the object tracking out we create 5 contexts, each of which > create 10 leaky objects before been thrown away, and: > > Tracked objects by class: > Config 1 > JavaScript::Context 1 > JavaScript::Runtime 1 > ThisObjectWillLeak 50 > ThisObjectWillNotLeak 5 > > :-( > > <cx_leaktest.pl>
Ok, I think I've found what causes this and it's the same problem as James Duncan has been experiencing. When an object is GC:ed in SpiderMonkey it calls PJS_finalize in PJS_Class.c. This function decreases the refcount so Perl has a chance to free the object too. Unfortunetly it did a SvREFCNT_dec on the RV itself and not the SV the RV references which is the cause of the leak. This seems to fix it: --- PJS_Class.c (revision 1969) +++ PJS_Class.c (local) @@ -172,7 +172,7 @@ void *ptr = JS_GetPrivate(cx, obj); if(ptr != NULL) { - SvREFCNT_dec((SV *) ptr); + SvREFCNT_dec(SvRV((SV *) ptr)); } } However now t/04-prototype.t doesn't pass anymore which is left as an exercise for me to fix. Cheerios! /Claes
Show quoted text
> Ok, I think I've found what causes this and it's the same problem as > James Duncan has been experiencing. When an object is GC:ed in > SpiderMonkey it calls PJS_finalize in PJS_Class.c. This function > decreases the refcount so Perl has a chance to free the object too.
Awesome! This fix and unbind_object have helped a lot. I'm curious; does SpiderMonkey call your PJS_finalize for simple return values, such as annonymous hash or arrayrefs?
Subject: Re: [rt.cpan.org #35571] JavaScript leaks memory like a sieve
Date: Fri, 2 May 2008 22:56:47 +0200
To: bug-JavaScript [...] rt.cpan.org
From: Claes Jakobsson <claes [...] versed.se>
On 2 maj 2008, at 22.30, CRAKRJACK via RT wrote: Show quoted text
> > Queue: JavaScript > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=35571 > > >
>> Ok, I think I've found what causes this and it's the same problem as >> James Duncan has been experiencing. When an object is GC:ed in >> SpiderMonkey it calls PJS_finalize in PJS_Class.c. This function >> decreases the refcount so Perl has a chance to free the object too.
> > Awesome! This fix and unbind_object have helped a lot. > > I'm curious; does SpiderMonkey call your PJS_finalize for simple > return > values, such as annonymous hash or arrayrefs?
No, anonymous hashes and arrays are converted between Perl and JS space instead of referenced as bound objects are. This means they do not belong to a JSClass instsance that we've declared and therefore does not defined PJS_finalize as the JSFinalizeOp. I think I'm going to do some check to see that we don't leak any memory there too just in case =) Cheers, Claes
CC: CRAKRJACK [...] cpan.org
Subject: Re: [rt.cpan.org #35571] JavaScript leaks memory like a sieve
Date: Fri, 02 May 2008 14:12:40 -0700
To: Claes Jakobsson via RT <bug-JavaScript [...] rt.cpan.org>
From: Tyler MacDonald <tyler [...] yi.org>
Claes Jakobsson via RT <bug-JavaScript@rt.cpan.org> wrote: Show quoted text
> No, anonymous hashes and arrays are converted between Perl and JS > space instead of referenced as bound objects are. This means they do > not belong to a JSClass instsance that we've declared and therefore > does not defined PJS_finalize as the JSFinalizeOp. > > I think I'm going to do some check to see that we don't leak any > memory there too just in case =)
I think there is still a leak somewhere, although it *definately* isnt as bad. The script I'm working on processes approx. 2,000,000 objects, and when it was skipping over binding them to javascript, it used about 800M by the end of it's run... before this fixes, it'd get to 3.7GB at around 800,000 items, then crash because the system only had 4GB available. Now, I'm at 572,000 items and 1019MB used. The script uses up 400MB before it even fires up the context, so that's ~600MB growth... still, that's a lot better than before :-) The main item is returned via bind_object/unbind_value; sub-items are returned as a set of objects in an arrayref returned from a bound method on the item object. The thing is, when I did the test with 100 items, neither items or their sub-objects leaked. So if there is a leak with the arrayrefs, the *contents* of them aren't leaking, which seems strange... - Tyler
From: CRAKRJACK [...] cpan.org
Show quoted text
> I think I'm going to do some check to see that we don't leak any > memory there too just in case =)
Claes, There's definately memory leaks somewhere. Until you/we're able to sort them out in the JavaScript module, I have had to resort to writing a glue module, that instead of linking to references, copies them. It's *very* purpose-specific: it only provides mechanisms to copy flat hashrefs containing only scalars, or arrayrefs of flat hashrefs. Here is the memory usage when bind_value and return values from methods connected with bind_class: at 100000 (508474 items/hr, 447M virt) at 200000 (546697 items/hr, 580M virt) at 300000 (572337 items/hr, 688M virt) at 400000 (555341 items/hr, 810M virt) at 500000 (616438 items/hr, 921M virt) at 600000 (579088 items/hr, 1052M virt) at 700000 (607375 items/hr, 1170M virt) at 800000 (644295 items/hr, 1286M virt) at 900000 (613171 items/hr, 1409M virt) at 1000000 (563027 items/hr, 1522M virt) Here is the memory usage using this braindead copying module: at 100000 (556414 items/hr, 358M virt) at 200000 (566483 items/hr, 364M virt) at 300000 (556701 items/hr, 372M virt) at 400000 (491635 items/hr, 381M virt) at 500000 (446982 items/hr, 386M virt) at 600000 (387374 items/hr, 396M virt) at 700000 (358056 items/hr, 400M virt) at 800000 (334300 items/hr, 411M virt) at 900000 (307458 items/hr, 420M virt) at 1000000 (282685 items/hr, 427M virt) I've attached the module... maybe something like this would be useful in JavaScript.xs anyways, such as "copy_value"? Data::Dumper etc would have some useful code to help deal with circular references. I guess you wouldn't be able to bind a perl class to a copied value, but you could still bind a JavaScript class to it, and it would certainly help for just passing data around in a way that doesnt make two garbage collectors fight with eachother. Cheers, Tyler
#include "EXTERN.h" #include "perl.h" #include "XSUB.h" #include "ppport.h" #include <jsapi.h> #include "../JavaScript-1.06_01/PJS_Context.h" char* get_key_name(HE* hashent) { SV* keysv; char* keyname; keysv = HeSVKEY(hashent); if (keysv) { /* great. Do nothing. */ warn ("here - got SV key %p", keysv); #ifdef JS_C_STRINGS_ARE_UTF8 keyname = SvPVutf8(keysv, SvLEN( keysv ) ); #else keyname = SvPVbyte(keysv, SvLEN( keysv ) ); #endif } else { /* otherwise, just a pv key */ keyname = HeKEY(hashent); #ifdef JS_C_STRINGS_ARE_UTF8 if (!HeKUTF8(hashent)) { /* key is bytes, we want utf8. */ keysv = newSV(0); sv_setpv( keysv, keyname ); keyname = SvPVutf8(keysv, SvLEN( keysv ) ); sv_2mortal( keysv ); } #else if (HeKUTF8(hashent)) { /* key is utf8, we want bytes. */ keysv = newSV(0); sv_setpv( keysv, keyname ); SvUTF8_on(keysv); keyname = SvPVbyte(keysv, SvLEN( keysv ) ); sv_2mortal( keysv ); } #endif } return keyname; } JSObject* import_perl_hash(HV* hash_in, JSContext* ctx) { HE* i; SV* sv_k; char* k; SV* v; I32* retlen; JSString* js_sv; jsdouble js_nv; jsval js_v; char *c_sv; JSObject* rv = JS_NewObject(ctx, NULL, NULL, NULL); hv_iterinit(hash_in); while((i = hv_iternext(hash_in)) != NULL) { k = get_key_name(i); v = (SV *)hv_iterval(hash_in, i); if(SvIOK(v) || SvNOK(v)) { js_nv = SvNV(v); JS_NewDoubleValue(ctx, js_nv, &js_v); } else if(SvPOK(v)) { STRLEN len; #ifdef JS_C_STRINGS_ARE_UTF8 c_sv = SvPVutf8(v, len); #else c_sv = SvPVbyte(v, len); #endif js_v = STRING_TO_JSVAL(JS_NewStringCopyN(ctx, c_sv, len)); } else { js_v = JSVAL_VOID; } JS_SetProperty(ctx, rv, k, &js_v); } return rv; } JSObject* import_perl_array_of_hashes(AV* ary, JSContext *ctx) { JSObject* rv = JS_NewArrayObject(ctx, 0, NULL); int i; SV** entry; jsval js_v; for(i=0;i<=av_len(ary);i++) { JSObject *jsentry; entry = av_fetch(ary, i, 0); if(entry) { jsentry = import_perl_hash((HV*)(SvRV(*(entry))), ctx); js_v = OBJECT_TO_JSVAL(jsentry); JS_SetElement(ctx, rv, i, &js_v); } else { printf("Entry %i does not exist!\n", i); } } return rv; } MODULE = Pricing::Repricer::Glue PACKAGE = Pricing::Repricer::Glue PROTOTYPES: DISABLE void send_hash_to_js(perl_cx, name, value) PJS_Context* perl_cx; const char* name; HV* value; PREINIT: JSObject* perlobj; JSObject* global; JSContext *cx; JSBool rv; jsval jsperlobj; CODE: cx = perl_cx->cx; perlobj = import_perl_hash(value, cx); jsperlobj = OBJECT_TO_JSVAL(perlobj); global = JS_GetGlobalObject(cx); JS_DeleteProperty(cx, global, name); rv = JS_SetProperty(cx, global, name, &jsperlobj); if(rv == JS_FALSE) die("Failed to assign javascript property!"); void send_array_of_hashes_to_js(perl_cx, name, value) PJS_Context* perl_cx; const char* name; AV* value; PREINIT: JSObject* perlobj; JSObject* global; JSContext *cx; JSBool rv; jsval jsperlobj; CODE: cx = perl_cx->cx; perlobj = import_perl_array_of_hashes(value, cx); jsperlobj = OBJECT_TO_JSVAL(perlobj); global = JS_GetGlobalObject(cx); JS_DeleteProperty(cx, global, name); rv = JS_SetProperty(cx, global, name, &jsperlobj); if(rv == JS_FALSE) die("Failed to assign javascript property!");