Bug #21242 for Cache-FastMmap: *** glibc detected *** double free or corruption (!prev): 0x00000000017e8d00 ***

Wed Aug 30 12:15:58 2006 melazyboy [...] gmail.com - Ticket created

Subject:

*** glibc detected *** double free or corruption (!prev): 0x00000000017e8d00 ***

On the following apache2 mpm configuration: root@nox:/var/log/apache2# apache2 -V Server version: Apache/2.0.55 Server built: Jul 26 2006 17:53:44 Server's Module Magic Number: 20020903:11 Architecture: 64-bit Server compiled with.... -D APACHE_MPM_DIR="server/mpm/worker" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT="" -D SUEXEC_BIN="/usr/lib/apache2/suexec2" -D DEFAULT_PIDLOG="/var/run/apache2.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types" -D SERVER_CONFIG_FILE="/etc/apache2/apache2.conf" I get the following error msg: root@nox:/var/log/apache2# /etc/init.d/apache2 restart * Forcing reload of apache 2.0 web server... *** glibc detected *** double free or corruption (!prev): 0x00000000017e9d80 *** /usr/sbin/apache2ctl: line 78: 19643 Aborted $HTTPD -k start -DSSL ...fail! The strace is attached as 'out' as follows. If there is *any* way i can assist you in solving this bug message me at evan@dealermade.com. This is all the info I knew to attach. Evan Carroll

Subject:

out

Download out
application/octet-stream 475.2k

Message body not shown because it is not plain text.

Mon Sep 25 20:35:26 2006 robm [...] fastmail.fm - Correspondence added

CC:	<evan [...] dealermade.com>
Subject:	Re: [rt.cpan.org #21242] * glibc detected * double free or corruption (!prev): 0x00000000017e8d00 ***
Date:	Tue, 26 Sep 2006 10:33:37 +1000
To:	<bug-Cache-FastMmap [...] rt.cpan.org>
From:	"Robert Mueller" <robm [...] fastmail.fm>

Hi I've been on vacation for a couple of weeks, so I'm only now back and have a chance to look at this. I haven't actually seen this problem before, and am not sure what would be causing it. A couple of things to check first: 1. I presume if you cache the session caching to something other that Cache::FastMmap, the problem goes away? 2. Can you get an ltrace rather than strace. That should list all the library calls which might help some more. Thanks Rob Show quoted text

> I get the following error msg: > root@nox:/var/log/apache2# /etc/init.d/apache2 restart > * Forcing reload of apache 2.0 web server... > *** glibc detected *** double free or corruption (!prev): > 0x00000000017e9d80 *** > /usr/sbin/apache2ctl: line 78: 19643 Aborted $HTTPD -k > start -DSSL > ...fail! > > > The strace is attached as 'out' as follows. > > If there is *any* way i can assist you in solving this bug message me at > evan@dealermade.com. This is all the info I knew to attach. > > Evan Carroll > >

Mon Sep 25 20:35:27 2006 The RT System itself - Status changed from 'new' to 'open'

Thu Sep 28 10:38:07 2006 melazyboy [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #21242] * glibc detected * double free or corruption (!prev): 0x00000000017e8d00 ***
Date:	Thu, 28 Sep 2006 09:37:44 -0500
To:	bug-Cache-FastMmap [...] rt.cpan.org
From:	"Evan Carroll" <melazyboy [...] gmail.com>

Message body is not shown because it is too large.

Mon Apr 02 04:46:56 2007 cpan [...] robm.fastmail.fm - Correspondence added

Replying to this from a while back. I'm not convinced this is Cache::FastMmaps fault in any way. From the ltrace there's no hint of Cache::FastMmap being loaded, it seems to occur immediately after something to do with mod_so apr_array_push(0x57a308, 0, 0x45a9d0, 10, 79) = 0x580d40 apr_array_push(0x57a638, 0, 0, 10, 0) = 0x580e08 strrchr("/build/buildd/apache2-2.0.55/bui"..., '/') = "/mod_so.c" strrchr("mod_so.c", '\\') = NULL apr_hook_sort_all(0, 0x57a138, 0xfefefefefefefeff, 0, 0xfefefefefefefeff <unfinished ...> --- SIGSEGV (Segmentation fault) --- +++ killed by SIGSEGV +++ I'll mark this as rejected for now. If you can still reproduce it and know it's related to Cache::FastMmap, please reopen the ticket. Rob

Mon Apr 02 04:46:58 2007 cpan [...] robm.fastmail.fm - Status changed from 'open' to 'rejected'

Mon Apr 02 23:20:03 2007 melazyboy [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #21242] * glibc detected * double free or corruption (!prev): 0x00000000017e8d00 ***
Date:	Mon, 2 Apr 2007 22:19:49 -0500
To:	bug-Cache-FastMmap [...] rt.cpan.org
From:	"Evan Carroll" <melazyboy [...] gmail.com>

It's Cache::FastMmap *related* because if I don't use Cache::FastMmap and use a different Catalyst store I get no error or crash. The condition is dependent on Cache::FastMmap. Now, whether or not Cache::FastMmap can place the blame on another component it uses is beyond me. The condition is still reproducible to date, on any Apache2-mpm-worker install w/ Cache::FastMmap and Catalyst. <wisper>A few Catalyst devs pointed the finger here.</wisper> Evan On 4/2/07, via RT <bug-Cache-FastMmap@rt.cpan.org> wrote: Show quoted text

> > > <URL: http://rt.cpan.org/Ticket/Display.html?id=21242 > > > Replying to this from a while back. I'm not convinced this is > Cache::FastMmaps fault in any way. From the ltrace there's no hint of > Cache::FastMmap being loaded, it seems to occur immediately after > something to do with mod_so > > apr_array_push(0x57a308, 0, 0x45a9d0, 10, 79) = 0x580d40 > apr_array_push(0x57a638, 0, 0, 10, 0) = 0x580e08 > strrchr("/build/buildd/apache2-2.0.55/bui"..., '/') = "/mod_so.c" > strrchr("mod_so.c", '\\') = NULL > apr_hook_sort_all(0, 0x57a138, 0xfefefefefefefeff, 0, 0xfefefefefefefeff > <unfinished ...> > --- SIGSEGV (Segmentation fault) --- > +++ killed by SIGSEGV +++ > > I'll mark this as rejected for now. If you can still reproduce it and > know it's related to Cache::FastMmap, please reopen the ticket. > > Rob > >

-- Evan Carroll System Lord of the Internets 832-445-8877

Mon Apr 02 23:20:07 2007 The RT System itself - Status changed from 'rejected' to 'open'

Tue Apr 03 07:52:07 2007 cpan [...] robm.fastmail.fm - Correspondence added

Show quoted text

> reproducible to date, on any Apache2-mpm-worker install w/ Cache::FastMmap

Ah, I just checked. I hadn't realised that mpm was a multi-process/multi-threaded hybrid. Cache::FastMmap does not natively support a multi-threaded environment at the moment. If you're so inclined, you could hack in your own CLONE sub so that it reopens the cache file anew in the cloned thread, but otherwise the current way perl clones objects when creating threads will result in a double free when it tries to clean up. If you have a patch to make it work I'll look at adding it back, but for now all I'm going to do is add a note in the next version that it's not thread safe. Rob

Tue Apr 03 18:54:22 2007 melazyboy [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #21242] * glibc detected * double free or corruption (!prev): 0x00000000017e8d00 ***
Date:	Tue, 3 Apr 2007 17:54:06 -0500
To:	bug-Cache-FastMmap [...] rt.cpan.org
From:	"Evan Carroll" <melazyboy [...] gmail.com>

Rob, Your repsonse is most apreciated. After having the "bug" open for such a long time, I'm glad it's acknolwedged that it isn't thread safe. I'm not sure with my knowledge of ithreads, I could write a patch, but I do know that after one is written many will rejoyce. Currently, this leaves the mod_perl Catalyst folks in a bind, using the higher perf. build of Apache2 results in a lower performance version of a Cache (file-based) while using the better cache requires the lower perf. build of Apache2. Unitl the day of the next version, I would suggest this be added as a makeshift or something of the like to eliminate any confusion on the end-user. sub CLONE { die __PACKAGE__ . 'does not support threads' } Thanks again, I look forward to the new version, Evan On 4/3/07, via RT <bug-Cache-FastMmap@rt.cpan.org> wrote: Show quoted text

> > > <URL: http://rt.cpan.org/Ticket/Display.html?id=21242 > >

> > reproducible to date, on any Apache2-mpm-worker install w/

> Cache::FastMmap > > Ah, I just checked. I hadn't realised that mpm was a > multi-process/multi-threaded hybrid. > > Cache::FastMmap does not natively support a multi-threaded environment > at the moment. If you're so inclined, you could hack in your own CLONE > sub so that it reopens the cache file anew in the cloned thread, but > otherwise the current way perl clones objects when creating threads will > result in a double free when it tries to clean up. > > If you have a patch to make it work I'll look at adding it back, but for > now all I'm going to do is add a note in the next version that it's not > thread safe. > > Rob > >

-- Evan Carroll System Lord of the Internets 832-445-8877

Tue May 08 07:35:05 2007 cpan [...] robm.fastmail.fm - Correspondence added

I've just released 1.15 which adds the die in the CLONE sub. That at least will stop people getting strange double free errors. I thought about threads support, but it's a bit of a pain. Also I'm not really convinced that perl threads support where everything has to be manually cloned makes the mixed process/thread model actually perform better? I would have thought with the work that has gone into most OS's with their COW forking and O(1) schedulers that the standard per-process model would be actually better. Is there some benchmarks which show that the mixed thread/process model does actually perform better?

Tue May 08 07:35:22 2007 cpan [...] robm.fastmail.fm - Status changed from 'open' to 'resolved'

Wed May 09 01:59:53 2007 melazyboy [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #21242] * glibc detected * double free or corruption (!prev): 0x00000000017e8d00 ***
Date:	Wed, 9 May 2007 00:58:51 -0500
To:	bug-Cache-FastMmap [...] rt.cpan.org
From:	"Evan Carroll" <melazyboy [...] gmail.com>

It is well accepted that the threaded apache model is better, and more flexible, whether there are advantages in perl is unknown by me. Though I concede to having been told by others that ithreads aren't worth the time for any advantage they might have over fork. Evan On 5/8/07, via RT <bug-Cache-FastMmap@rt.cpan.org> wrote: Show quoted text

> > > <URL: http://rt.cpan.org/Ticket/Display.html?id=21242 > > > I've just released 1.15 which adds the die in the CLONE sub. That at > least will stop people getting strange double free errors. > > I thought about threads support, but it's a bit of a pain. Also I'm not > really convinced that perl threads support where everything has to be > manually cloned makes the mixed process/thread model actually perform > better? I would have thought with the work that has gone into most OS's > with their COW forking and O(1) schedulers that the standard per-process > model would be actually better. Is there some benchmarks which show that > the mixed thread/process model does actually perform better? >

-- Evan Carroll System Lord of the Internets 832-445-8877

Wed May 09 01:59:56 2007 The RT System itself - Status changed from 'resolved' to 'open'

Thu May 10 02:43:54 2007 cpan [...] robm.fastmail.fm - Correspondence added

Because this bug has a cryptic subject and information that's not relevant, I've created a new ticket related to threads support and I'm closing this one. http://rt.cpan.org/Ticket/Display.html?id=27050

Thu May 10 02:43:56 2007 cpan [...] robm.fastmail.fm - Status changed from 'open' to 'resolved'