Bug #63458 for Class-XSAccessor: segfault in C:XSA, perl 5.12.2, linux/amd64

Mon Nov 29 21:19:58 2010 TJC [...] cpan.org - Ticket created

Subject:

segfault in C:XSA, perl 5.12.2, linux/amd64

While tracking down test failures in some modules that use DBIx::Class, I discovered the cause was segfaults from Class::XSAccessor. For eg, HTML::FormHandler::Model::DBIC fails. Manually running the test script (ie. perl -Ilib t/book.t) results in the following occuring, after code gets into the END {} block. *** glibc detected *** perl: realloc(): invalid pointer: 0x0000000001674838 *** ======= Backtrace: ========= /lib/libc.so.6(+0x775b6)[0x7f46229f65b6] /lib/libc.so.6(realloc+0x352)[0x7f46229fd312] /usr/local/strategic/perl/lib/site_perl/5.12.2/x86_64-linux-thread- multi/auto/Class/XSAccessor/XSAccessor.so(get_hashkey_index+0x2a7)[0x7f46210dcb77] /usr/local/strategic/perl/lib/site_perl/5.12.2/x86_64-linux-thread- multi/auto/Class/XSAccessor/XSAccessor.so(XS_Class__XSAccessor_newxs_accessor+0x1db)[0x7f46210dd5cb] perl(Perl_pp_entersub+0x530)[0x495c90] perl(Perl_runops_standard+0x16)[0x4947d6] perl(Perl_call_sv+0x4b7)[0x439a47] perl(Perl_call_list+0x2ad)[0x439f1d] perl(perl_destruct+0x170c)[0x43c3bc] perl(main+0xcb)[0x421e7b] /lib/libc.so.6(__libc_start_main+0xfd)[0x7f462299dc4d] perl[0x421ce9] ======= Memory map: ======== 00400000-0055c000 r-xp 00000000 fb:03 6828319 /home/tobyc/git/modern- perl/system/lucid-amd64.new/bin/perl 0075b000-0075c000 r--p 0015b000 fb:03 6828319 /home/tobyc/git/modern- perl/system/lucid-amd64.new/bin/perl 0075c000-00760000 rw-p 0015c000 fb:03 6828319 /home/tobyc/git/modern- perl/system/lucid-amd64.new/bin/perl 01674000-03e44000 rw-p 00000000 00:00 0 [heap] 7f461f7c8000-7f461f7de000 r-xp 00000000 fb:00 29039 /lib/libgcc_s.so.1 7f461f7de000-7f461f9dd000 ---p 00016000 fb:00 29039 /lib/libgcc_s.so.1 7f461f9dd000-7f461f9de000 r--p 00015000 fb:00 29039 /lib/libgcc_s.so.1 7f461f9de000-7f461f9df000 rw-p 00016000 fb:00 29039 /lib/libgcc_s.so.1 7f461f9df000-7f461f9e0000 ---p 00000000 00:00 0 7f461f9e0000-7f46201e0000 rw-p 00000000 00:00 0 7f46201e0000-7f462027c000 r-xp 00000000 fb:03 5914083 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/site_perl/5.12.2/x86_64-linux-thread-multi/auto/DBD/SQLite/SQLite.so 7f462027c000-7f462047c000 ---p 0009c000 fb:03 5914083 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/site_perl/5.12.2/x86_64-linux-thread-multi/auto/DBD/SQLite/SQLite.so 7f462047c000-7f462047e000 r--p 0009c000 fb:03 5914083 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/site_perl/5.12.2/x86_64-linux-thread-multi/auto/DBD/SQLite/SQLite.so 7f462047e000-7f4620480000 rw-p 0009e000 fb:03 5914083 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/site_perl/5.12.2/x86_64-linux-thread-multi/auto/DBD/SQLite/SQLite.so 7f4620480000-7f4620481000 rw-p 00000000 00:00 0 7f4620481000-7f462048a000 r-xp 00000000 fb:03 26325 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so 7f462048a000-7f4620689000 ---p 00009000 fb:03 26325 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so 7f4620689000-7f462068a000 r--p 00008000 fb:03 26325 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so 7f462068a000-7f462068b000 rw-p 00009000 fb:03 26325 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Data/Dumper/Dumper.so 7f462068b000-7f46206ac000 r-xp 00000000 fb:03 5915678 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/site_perl/5.12.2/x86_64-linux-thread-multi/auto/DBI/DBI.so 7f46206ac000-7f46208ab000 ---p 00021000 fb:03 5915678 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/site_perl/5.12.2/x86_64-linux-thread-multi/auto/DBI/DBI.so 7f46208ab000-7f46208ac000 r--p 00020000 fb:03 5915678 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/site_perl/5.12.2/x86_64-linux-thread-multi/auto/DBI/DBI.so 7f46208ac000-7f46208ad000 rw-p 00021000 fb:03 5915678 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/site_perl/5.12.2/x86_64-linux-thread-multi/auto/DBI/DBI.so 7f46208ad000-7f46208b0000 r-xp 00000000 fb:03 26231 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so 7f46208b0000-7f4620aaf000 ---p 00003000 fb:03 26231 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so 7f4620aaf000-7f4620ab0000 r--p 00002000 fb:03 26231 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so 7f4620ab0000-7f4620ab1000 rw-p 00003000 fb:03 26231 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Cwd/Cwd.so 7f4620ab1000-7f4620ac5000 r-xp 00000000 fb:03 26320 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Storable/Storable.so 7f4620ac5000-7f4620cc5000 ---p 00014000 fb:03 26320 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Storable/Storable.so 7f4620cc5000-7f4620cc6000 r--p 00014000 fb:03 26320 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Storable/Storable.so 7f4620cc6000-7f4620cc7000 rw-p 00015000 fb:03 26320 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Storable/Storable.so 7f4620cc7000-7f4620cca000 r-xp 00000000 fb:03 26294 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so 7f4620cca000-7f4620eca000 ---p 00003000 fb:03 26294 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so 7f4620eca000-7f4620ecb000 r--p 00003000 fb:03 26294 /home/tobyc/git/modern- perl/system/lucid-amd64.new/lib/5.12.2/x86_64-linux-thread-multi/auto/Fcntl/Fcntl.so Aborted

Mon Nov 29 21:20:36 2010 TJC [...] cpan.org - Dependency by ticket #63455 added

Mon Nov 29 21:22:22 2010 TJC [...] cpan.org - Requestor toby.corkindale@strategicdata.com.au added

Mon Nov 29 21:31:56 2010 TJC [...] cpan.org - Correspondence added

I don't know if it helps to note that the segfault occurred while Perl was processing the END {} block - would some parts of memory have already been cleared down by then?

Mon Nov 29 21:32:02 2010 TJC [...] cpan.org - Status changed from 'new' to 'open'

Mon Nov 29 21:41:10 2010 TJC [...] cpan.org - Dependency by ticket #63459 added

Tue Nov 30 07:22:16 2010 smueller [...] cpan.org - Correspondence added

I'm currently installing the modules in question. Not sure if I'll have time to dig in later today. Thanks for reporting this with a test case that I may actually be able to use for reproducing the bug. I hope that'll lead to a "doh" moment as soon as I get things running in gdb. Cheers, Steffen

Tue Nov 30 08:12:49 2010 smueller [...] cpan.org - Correspondence added

On Mon Nov 29 21:19:58 2010, TJC wrote: Show quoted text

> For eg, HTML::FormHandler::Model::DBIC fails. > > Manually running the test script (ie. perl -Ilib t/book.t) results in > the following occuring, after code > gets into the END {} block.

Curiously, those tests fail for me in the same way (without segfaults) regardless of whether Class::XSAccessor is available or not. I'm on OSX, though. I'll try to get access to a saner (i.e. Linux) system for testing. In the meantime: What version of Class::XSAccessor were you using? Could you try with: - The most recent release - Version 1.06 - github master? Best regards, Steffen

Tue Nov 30 08:46:48 2010 TJC [...] cpan.org - Correspondence added

On Tue Nov 30 08:12:49 2010, SMUELLER wrote: Show quoted text

> On Mon Nov 29 21:19:58 2010, TJC wrote:

> > For eg, HTML::FormHandler::Model::DBIC fails. > > > > Manually running the test script (ie. perl -Ilib t/book.t) results

> in

> > the following occuring, after code > > gets into the END {} block.

> > Curiously, those tests fail for me in the same way (without segfaults) > regardless of whether > Class::XSAccessor is available or not. I'm on OSX, though.

The tests are a bit delicate; they rely upon a pre-existing SQLite database, and the tests are supposed to leave in a consistent state, but don't always seem to. You can recreate the DB by deleting t/db/book.db, then running sqlite3 t/db/book.db ".read t/db/bookdb.sql" Although I noticed that 'prove' was hiding the segfaults for me, and just reporting '0 tests failed' but with 'non-zero exit code' or something. It wasn't until I ran "perl -Ilib t/book.t" that I saw the segfault output. Show quoted text

> I'll try to get access to a saner (i.e. Linux) system for testing.

If it helps, the system I'm having trouble with is Debian Lenny, amd64 architecture, although we have a later Perl (5.12.2) rather than the system perl (5.10.1). I can provide either the exact commands used to roll it, or a tarball containing the self- contained Perl + modules that demonstrate the problem, if it would help. ie. That's suitable for running in a chroot/lxc/jail kind of environment. Show quoted text

> In the meantime: What version of Class::XSAccessor were you using?

It was the latest version on CPAN as of about 12 hours ago, so 1.09 I think. I'll confirm in the morning (umm, about 9 hours from now). Show quoted text

> Could you try with: > > - The most recent release > - Version 1.06 > - github master?

I need some sleep here, will try the older version, and trunk version, in the morning. Thanks for looking into this, Toby

Tue Nov 30 15:32:41 2010 smueller [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #63458] segfault in C:XSA, perl 5.12.2, linux/amd64
Date:	Tue, 30 Nov 2010 21:32:34 +0100
To:	bug-Class-XSAccessor [...] rt.cpan.org
From:	Steffen Mueller <smueller [...] cpan.org>

On Nov 30, 2010, at 2:46 PM, Toby C via RT wrote: Show quoted text

> The tests are a bit delicate; they rely upon a pre-existing SQLite database, and the tests are > supposed to leave in a consistent state, but don't always seem to. > You can recreate the DB by deleting t/db/book.db, then running > sqlite3 t/db/book.db ".read t/db/bookdb.sql" > > Although I noticed that 'prove' was hiding the segfaults for me, and just reporting '0 tests > failed' but with 'non-zero exit code' or something. > It wasn't until I ran "perl -Ilib t/book.t" that I saw the segfault output.

I can reproduce it on OS X now. #0 0x000000010075744c in get_hashkey_index () #1 0x0000000100758208 in XS_Class__XSAccessor_newxs_accessor () #2 0x000000010009f033 in Perl_pp_entersub () #3 0x0000000100096a2a in Perl_runops_standard () #4 0x000000010001cd4a in Perl_call_sv () #5 0x000000010001d273 in Perl_call_list () #6 0x000000010001ed1e in perl_destruct () #7 0x00000001000012b2 in main () If I do this: --- a/XSAccessor.xs +++ b/XSAccessor.xs @@ -563,7 +563,7 @@ END() PROTOTYPE: CODE: if (CXSAccessor_reverse_hashkeys) { - CXSA_HashTable_free(CXSAccessor_reverse_hashkeys); + /*CXSA_HashTable_free(CXSAccessor_reverse_hashkeys);*/ } void Then it doesn't segfault any more. This is essentially neglecting to free memory at the end of the process. Now, let me guess why this would happen: a) There is a class with a DESTROY method that calls XS accessors. b) Instances of that class get destroyed after this particular END block runs. I'm not sure how we can tell perl that we really, really want to run really late. But then there would be contention at THAT time at some point. Or maybe get_hashkey_index needs to be made aware of the status that things have been freed already. Except what kind of behaviour would that introduce in Perl land? "Sorry, can't properly DESTROY your object. We've already shut down the code that does it!" I think our only option is to leak the memory. Chocolateboy, what do you think? Cheers, Steffen

Wed Dec 01 14:28:06 2010 smueller [...] cpan.org - Correspondence added

On Tue Nov 30 15:32:41 2010, SMUELLER wrote: Show quoted text

> I think our only option is to leak the memory.

... and I still haven't come up with a better solution. I will try to make a release that relies on the OS to free the memory (that would otherwise be freed during END) later today to fix the crashing bug. If we come up with something better, that's the next iteration. The only case I can think of in which this really is an issue is long running applications that repeatedly create and destroy embedded perl interpreters. Since I have a hunch that that would a) be rare and b) leak memory anyway, I'm not that worried. Cheers, Steffen

Wed Dec 01 14:59:31 2010 smueller [...] cpan.org - Correspondence added

Class::XSAccessor 1.10 released to CPAN. It would be great if you could confirm that the problem went away. I'll keep the ticket open until then. --Steffen

Wed Dec 01 14:59:32 2010 smueller [...] cpan.org - Status changed from 'open' to 'patched'

Wed Dec 01 20:39:17 2010 TJC [...] cpan.org - Correspondence added

On Wed Dec 01 14:59:31 2010, SMUELLER wrote: Show quoted text

> Class::XSAccessor 1.10 released to CPAN. It would be great if you > could confirm that the > problem went away. > > I'll keep the ticket open until then.

My specific instance seems to be fixed with 1.10 - thanks for your rapid work on this. I'm a little nervous at the mention of memory leaks; could we discuss that a little further? I see two possible cases, and I'm not sure which it is. Could you take a moment to explain which case it is? 1) Memory will only be leaked when Perl is "exiting". As I understand it, this is OK for shell script and long-running instances, but not good for things like mod_perl or postgres' PL/Perl. 2) Memory will only be leaked on object destruction - ie. when a blessed object goes out of scope. I ask because I see there was discussed of DESTROY methods. Thanks, Toby

Wed Dec 01 20:39:18 2010 The RT System itself - Status changed from 'patched' to 'open'

Thu Dec 02 02:28:30 2010 smueller [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #63458] segfault in C:XSA, perl 5.12.2, linux/amd64
Date:	Thu, 2 Dec 2010 08:28:19 +0100
To:	bug-Class-XSAccessor [...] rt.cpan.org
From:	Steffen Mueller <smueller [...] cpan.org>

On Dec 2, 2010, at 2:39 AM, Toby C via RT wrote: Show quoted text

> Queue: Class-XSAccessor > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=63458 > > > On Wed Dec 01 14:59:31 2010, SMUELLER wrote:

>> Class::XSAccessor 1.10 released to CPAN. It would be great if you >> could confirm that the >> problem went away. >> >> I'll keep the ticket open until then.

Show quoted text

> I see two possible cases, and I'm not sure which it is. Could you take a > moment to explain which case it is? > > 1) Memory will only be leaked when Perl is "exiting". As I understand > it, this is OK for shell script and long-running instances, but not good > for things like mod_perl or postgres' PL/Perl.

This is the case. But I highly doubt that this matters for mod_perl. mod_perl instances are forked, are they not? They're not many interpreters in the same process. For Postgres, I'm not sure. But seriously. I am very certain that a lot of modules rely on global destruction for this cleanup to happen, so Postgres probably doesn't do this either. And this memory might (and it's early in the morning, I'm not sure) even be reused by new instances that have the same hash key names. Show quoted text

> 2) Memory will only be leaked on object destruction - ie. when a blessed > object goes out of scope. I ask because I see there was discussed of > DESTROY methods.

This is not the case. --Steffen

Thu Dec 02 02:34:48 2010 TJC [...] cpan.org - Correspondence added

On Thu Dec 02 02:28:30 2010, SMUELLER wrote: Show quoted text

> On Dec 2, 2010, at 2:39 AM, Toby C via RT wrote:

> > I see two possible cases, and I'm not sure which it is. Could you > > take a moment to explain which case it is? > > > > 1) Memory will only be leaked when Perl is "exiting". As I > > understand it, this is OK for shell script and long-running > > instances, but not good for things like mod_perl or > > postgres' PL/Perl.

> > This is the case. > But I highly doubt that this matters for mod_perl. mod_perl instances > are forked, are they not? They're not many interpreters in the same > process. For Postgres, I'm not sure. But seriously. I am very > certain that a lot of modules rely on global destruction for this > cleanup to happen, so Postgres probably doesn't do this either. And > this memory might (and it's early in the morning, I'm not sure) > even be reused by new instances that have the same hash key names.

mod_perl is threaded if you run Apache with the "mpm_worker" engine (ie. threaded Apache), and I think that's also the only option on win32. In those cases, you have a pool of Perl interpreter threads that handle many requests. (That's probably why the win32+mod_perl environment is a bit fragile; any memory access mistakes from one thread can effect either another thread, or a later request) That said - I don't think many people tend to use mod_perl in the threaded apache environment on Linux, since it does tend to be unstable due to exactly those sort of issues.. Show quoted text

> > 2) Memory will only be leaked on object destruction - ie. when a

> blessed

> > object goes out of scope. I ask because I see there was discussed of > > DESTROY methods.

> > This is not the case.

That's good - thanks for reassuring me. cheers, Toby

Thu Dec 02 13:12:41 2010 smueller [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #63458] segfault in C:XSA, perl 5.12.2, linux/amd64
Date:	Thu, 2 Dec 2010 19:12:33 +0100
To:	bug-Class-XSAccessor [...] rt.cpan.org
From:	Steffen Mueller <smueller [...] cpan.org>

On Dec 2, 2010, at 8:34 AM, Toby C via RT wrote: Show quoted text

> mod_perl is threaded if you run Apache with the "mpm_worker" engine (ie. threaded Apache), > and I think that's also the only option on win32. In those cases, you have a pool of Perl > interpreter threads that handle many requests.

Not sure how mod_perl implements this, but having many ithreads and repeatedly creating/reaping them is okay as far as C::XSA goes. Just destroying the master thread means leaking the memory. --Steffen

Thu Dec 02 13:27:30 2010 smueller [...] cpan.org - Correspondence added

Resolved! Yay!

Thu Dec 02 13:27:32 2010 smueller [...] cpan.org - Status changed from 'open' to 'resolved'