Bug #26411 for JavaScript: undefined symbol: js_InternalInvoke when using libmozjs from xulrunner

Mon Apr 16 15:27:50 2007 ntyni [...] iki.fi - Ticket created

Subject:

undefined symbol: js_InternalInvoke when using libmozjs from xulrunner

Hi, I'm packaging the Javascript module for Debian, and there's a linker error with 1.03 when running 'make test': t/00-init......................... # Failed test 'use JavaScript;' # in t/00-init.t at line 5. # Tried to use 'JavaScript'. # Error: Can't load '/tmp/buildd/libjavascript-perl-1.03/blib/arch/auto/JavaScript/JavaScript.so' for module JavaScript: /tmp/buildd/libjavascript-perl-1.03/blib/arch/auto/JavaScript/JavaScript.so: undefined symbol: js_InternalInvoke at /usr/lib/perl/5.8/DynaLoader.pm line 225. I have looked into this, and the problem as I understand it is that Javascript.xs is using js_InternalCall(), which is not part of the public API of libmozjs. (The js_InternalCall() function is #define'd as js_InternalInvoke() in <mozjs/js_interph>, hence the linker error message.) Now, the Debian libmozjs comes from the xulrunner package: http://ftp.mozilla.org/pub/mozilla.org/xulrunner/ which contains a copy of the same code as ftp://ftp.mozilla.org/pub/js but with one notable difference: the xulrunner build system hides all library symbols by default, unless they are declared as public. This comes from config/gcc_hidden.h in the xulrunner package: #pragma GCC visibility push(hidden) which is more or less the same as compiling with "gcc -fvisibility=hidden". Since js_InternalCall() (or js_InternalInvoke()) is not a public symbol, it's not exported from libmozjs.so (as seen with eg. objdump -T) and the result is the linker error. I don't know much about mozjs internals, but it seems to me that the right fix is to use only the symbols declared as JS_PUBLIC_API (or maybe JS_FRIEND_API, I don't know) in the headers. Version numbers: this has been researched with xulrunner 1.8.0.11, which is apparently not an official release but just a release tag in the mozilla CVS repository. I did look at 1.8.0.4, which is available as an official released tarball, and I don't see any difference in this regard. I haven't actually tested this against anything else than 1.8.0.11, though. The mozjs version inside xulrunner 1.8.0.11 seems to be 1.60, according to jsconfig.h . Hope this makes sense. Please let me know if you need more information, and I'll do my best to help. Thanks for your work on the Javascript module, -- Niko Tyni (on behalf of the Debian Perl Group) ntyni@iki.fi

Tue Apr 17 10:26:43 2007 CLAESJAC [...] cpan.org - Correspondence added

Hi, I have checked in a new version into my repository that does not call js_InternalCall but instead uses JS_CallFunction. Could you please do a checkout of that and test against it. The one currently in trunk/ will be released as 1.04 very soon. The code can be checked out from svn:// svn.surfar.nu/public/JavaScript/trunk If it works please let me know and I can close this bug when I make the release. I'm also interested to know how your xulrunner/libmozjs is build.. Is unicode and e4x enabled? is it compiled with JS_THREADSAFE? Thanks, Claes

Tue Apr 17 10:26:49 2007 The RT System itself - Status changed from 'new' to 'open'

Tue Apr 17 10:26:59 2007 CLAESJAC [...] cpan.org - Given to CLAESJAC

Tue Apr 17 14:15:55 2007 ntyni [...] iki.fi - Correspondence added

Subject:	Re: [rt.cpan.org #26411] undefined symbol: js_InternalInvoke when using libmozjs from xulrunner
Date:	Tue, 17 Apr 2007 21:14:11 +0300
To:	Claes Jakobsson via RT <bug-JavaScript [...] rt.cpan.org>
From:	Niko Tyni <ntyni [...] iki.fi>

On Tue, Apr 17, 2007 at 10:26:52AM -0400, Claes Jakobsson via RT wrote: Show quoted text

> I have checked in a new version into my repository that does not call js_InternalCall but instead > uses JS_CallFunction. Could you please do a checkout of that and test against it. The one > currently in trunk/ will be released as 1.04 very soon. The code can be checked out from svn:// > svn.surfar.nu/public/JavaScript/trunk

There's a typo in exceptions.c, and JavaScript.xs is still using js_Execute(), which is another non-public symbol. I'm attaching a patch that fixes these for me. Furthermore, if configuring with JS_ENABLE_E4X, there's one more unknown symbol: js_ValueToXMLString . I can't see an easy fix for this. Without JS_ENABLE_E4X and JS_C_STRINGS_ARE_UTF8, most of the tests run correctly with the patch, but there are still some failures: t/02-types-from-perl.t 1 256 18 1 5.56% 6 t/29-exceptions.t 9 2304 17 9 52.94% 2-4 6 9 12 15-17 3 tests skipped. Failed 2/32 test scripts, 93.75% okay. 10/238 subtests failed, 95.80% okay. I'm attaching the full log. Please let me know if I can do something to help with these. Show quoted text

> I'm also interested to know how your xulrunner/libmozjs is build.. Is

unicode and e4x enabled? > is it compiled with JS_THREADSAFE? JS_THREADSAFE is enabled, JS_C_STRINGS_ARE_UTF8 doesn't seem to be (t/23-unicode.t fails noisily if I answer "yes" here.) I'm not sure about the E4X stuff; looking at the js code in xulrunner JS_HAS_XML_SUPPORT seems to be enabled automatically when JS_VERSION==160. The tests fail if I enable this because of the js_ValueToXMLString problem. BTW, there's a function in the public API for testing JS_C_STRINGS_ARE_UTF8: JS_StringsAreUTF8(). Maybe you could use this instead of hardcoding it at build time? (I can file a wishlist bug about this if you prefer to discuss it separately.) One more thing: Debian puts the nspr headers in /usr/include/nspr, and <jslock.h> includes <pratom.h> and others. The patch adds a check for this in Makefile.PL too. Cheers, -- Niko Tyni ntyni@iki.fi

Message body is not shown because sender requested not to inline it.

Tue Apr 17 15:31:09 2007 claes [...] surfar.nu - Correspondence added

Subject:	Re: [rt.cpan.org #26411] undefined symbol: js_InternalInvoke when using libmozjs from xulrunner
Date:	Tue, 17 Apr 2007 21:30:33 +0200
To:	bug-JavaScript [...] rt.cpan.org
From:	Claes Jakobsson <claes [...] surfar.nu>

Hi, On 17 apr 2007, at 20.16, ntyni@iki.fi via RT wrote: Show quoted text

> > There's a typo in exceptions.c, and JavaScript.xs is still using > js_Execute(), which is another non-public symbol. I'm attaching a > patch > that fixes these for me.

Your patch has been integrated and I've commited it to my repository. Show quoted text

> Furthermore, if configuring with JS_ENABLE_E4X, there's one more > unknown > symbol: js_ValueToXMLString . I can't see an easy fix for this.

I'll have a look at this and see how it can be fixed. Show quoted text

> Without JS_ENABLE_E4X and JS_C_STRINGS_ARE_UTF8, most of the tests > run correctly with the patch, but there are still some failures: > > t/02-types-from-perl.t 1 256 18 1 5.56% 6 > t/29-exceptions.t 9 2304 17 9 52.94% 2-4 6 9 12 15-17 > 3 tests skipped. > Failed 2/32 test scripts, 93.75% okay. 10/238 subtests failed, > 95.80% okay. > > I'm attaching the full log. Please let me know if I can do > something to > help with these.

Hm... are you running this on a 64-bit machine? I recall there's been some problems with 02-types-from-perl.t test on 64-bit machines and/ or Perl or SpiderMonkey compiled with support for 64-bit integers. Why the exceptions tests would fail I have no idea. I'll try to download and build the JavaScript engine your are using and run the tests on that or even better if I could be provided with shell access to a machine you're building and running on so I can test there? Show quoted text

>> I'm also interested to know how your xulrunner/libmozjs is build.. Is

> unicode and e4x enabled? > is it compiled with JS_THREADSAFE? > > JS_THREADSAFE is enabled, JS_C_STRINGS_ARE_UTF8 doesn't seem to be > (t/23-unicode.t fails noisily if I answer "yes" here.) > > I'm not sure about the E4X stuff; looking at the js code in xulrunner > JS_HAS_XML_SUPPORT seems to be enabled automatically when > JS_VERSION==160. > The tests fail if I enable this because of the js_ValueToXMLString > problem. > > BTW, there's a function in the public API for testing > JS_C_STRINGS_ARE_UTF8: JS_StringsAreUTF8(). Maybe you could use this > instead of hardcoding it at build time? (I can file a wishlist bug > about this if you prefer to discuss it separately.)

Yeah, I'm aware of JS_StringsAreUTF8 but unfortunately that function is not available in SpiderMonkey 1.5 or earlier. Support for older (1.5 and previous) will probably be dropped soon so by then I can switch to runtime detection. Thanks, Claes

Wed Apr 18 06:49:16 2007 ntyni [...] iki.fi - Correspondence added

Subject:	Re: [rt.cpan.org #26411] undefined symbol: js_InternalInvoke when using libmozjs from xulrunner
Date:	Wed, 18 Apr 2007 13:48:27 +0300
To:	"claes [...] surfar.nu via RT" <bug-JavaScript [...] rt.cpan.org>
From:	Niko Tyni <ntyni [...] iki.fi>

On Tue, Apr 17, 2007 at 03:31:11PM -0400, claes@surfar.nu via RT wrote: Show quoted text

> > t/02-types-from-perl.t 1 256 18 1 5.56% 6 > > t/29-exceptions.t 9 2304 17 9 52.94% 2-4 6 9 12 15-17 > > 3 tests skipped. > > Failed 2/32 test scripts, 93.75% okay. 10/238 subtests failed, > > 95.80% okay. > > > > I'm attaching the full log. Please let me know if I can do > > something to > > help with these.

> > Hm... are you running this on a 64-bit machine? I recall there's been > some problems with 02-types-from-perl.t test on 64-bit machines and/ > or Perl or SpiderMonkey compiled with support for 64-bit integers.

Yes, it's a 64-bit (x86_64) machine. Sorry for not mentioning this earlier. The test indeed succeeds on a 32-bit x86. I'll look further into this later. Show quoted text

> Why the exceptions tests would fail I have no idea.

It's a side effect from the js_Execute -> JS_ExecuteScript switch. I have debugged it quite a bit, and JS_ExecuteScript() checks for unhandled exceptions after doing js_Execute() (LAST_FRAME_CHECKS -> js_ReportUncaughtException()) and clears them after calling the error handler. So it looks like this needs a default error handler, otherwise the exception is lost. No working patch yet, though :( I would expect this to be reproducible for you too after the JS_ExecuteScript change... is the test really working for you? Show quoted text

> I'll try to > download and build the JavaScript engine your are using and run the > tests on that or even better if I could be provided with shell access > to a machine you're building and running on so I can test there?

Yes, I guess I can arrange that. I'll get back to this in a private email. Cheers, -- Niko Tyni ntyni@iki.fi

Wed Apr 18 07:14:34 2007 claes [...] surfar.nu - Correspondence added

Subject:	Re: [rt.cpan.org #26411] undefined symbol: js_InternalInvoke when using libmozjs from xulrunner
Date:	Wed, 18 Apr 2007 13:12:41 +0200
To:	bug-JavaScript [...] rt.cpan.org
From:	Claes Jakobsson <claes [...] surfar.nu>

Hi, On 18 apr 2007, at 12.49, ntyni@iki.fi via RT wrote: Show quoted text

> Yes, it's a 64-bit (x86_64) machine. Sorry for not mentioning this > earlier. The test indeed succeeds on a 32-bit x86. I'll look further > into this later.

That's what I thought. Show quoted text

>> Why the exceptions tests would fail I have no idea.

> > It's a side effect from the js_Execute -> JS_ExecuteScript switch. > I have debugged it quite a bit, and JS_ExecuteScript() checks for > unhandled exceptions after doing js_Execute() (LAST_FRAME_CHECKS -> > js_ReportUncaughtException()) and clears them after calling the error > handler. So it looks like this needs a default error handler, > otherwise > the exception is lost. No working patch yet, though :( > > I would expect this to be reproducible for you too after the > JS_ExecuteScript change... is the test really working for you?

I've been able to reproduce this now and I'm looking at how to fix it. I was running my tests against a SM 1.7 engine which has JSOPTION_DONT_REPORT_UNCAUGHT that prevents exceptions to be converted to error reports that will be taken care of by the error handler. Thanks, Claes

Thu Apr 19 14:17:36 2007 claes [...] surfar.nu - Correspondence added

Subject:	Re: [rt.cpan.org #26411] undefined symbol: js_InternalInvoke when using libmozjs from xulrunner
Date:	Thu, 19 Apr 2007 20:17:03 +0200
To:	bug-JavaScript [...] rt.cpan.org
From:	Claes Jakobsson <claes [...] surfar.nu>

Hi, On 17 apr 2007, at 20.16, ntyni@iki.fi via RT wrote: Show quoted text

> Furthermore, if configuring with JS_ENABLE_E4X, there's one more > unknown > symbol: js_ValueToXMLString . I can't see an easy fix for this.

This should now be fixed in my SVN version. Could you please do a test and verify it works in your environment. Thanks, Claes

Fri Apr 20 03:19:15 2007 ntyni [...] iki.fi - Correspondence added

Subject:	Re: [rt.cpan.org #26411] undefined symbol: js_InternalInvoke when using libmozjs from xulrunner
Date:	Fri, 20 Apr 2007 10:18:47 +0300
To:	"claes [...] surfar.nu via RT" <bug-JavaScript [...] rt.cpan.org>
From:	Niko Tyni <ntyni [...] iki.fi>

On Thu, Apr 19, 2007 at 02:17:39PM -0400, claes@surfar.nu via RT wrote: Show quoted text

> On 17 apr 2007, at 20.16, ntyni@iki.fi via RT wrote:

> > Furthermore, if configuring with JS_ENABLE_E4X, there's one more > > unknown > > symbol: js_ValueToXMLString . I can't see an easy fix for this.

> > This should now be fixed in my SVN version. Could you please do a > test and verify it works in your environment.

Yes, there are no missing symbols anymore and the tests in t/26-e4x.t (and all others except t/29-exceptions.t) now succeed. Thanks, -- Niko Tyni ntyni@iki.fi

Sun Oct 28 17:09:04 2007 ntyni [...] iki.fi - Correspondence added

From:

ntyni [...] iki.fi

On Fri Apr 20 03:19:15 2007, ntyni@iki.fi wrote: Show quoted text

> Yes, there are no missing symbols anymore and the tests in t/26-e4x.t > (and all others except t/29-exceptions.t) now succeed.

Hi, it looks like time has solved this problem: the last test failures in t/29-exceptions.t are now gone when building with SpiderMonkey 1.70. Presumably this was a bug in 1.60 then; I haven't investigated more closely. Any chance of a release of the current SVN code? Or shall we get a snapshot into Debian for now? Cheers, -- Niko Tyni ntyni@iki.fi

Mon Nov 19 10:44:58 2007 CLAESJAC [...] cpan.org - Status changed from 'open' to 'resolved'

Mon Nov 19 10:45:08 2007 CLAESJAC [...] cpan.org - Fixed in 1.04 added