Skip Menu |

This queue is for tickets about the threads CPAN distribution.

Report information
The Basics
Id: 35836
Status: resolved
Priority: 0/
Queue: threads

People
Owner: Nobody in particular
Requestors: smueller [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in:
  • 1.57
  • 1.58
  • 1.59
  • 1.61
  • 1.62
  • 1.63
  • 1.64
  • 1.65
  • 1.66
  • 1.67
  • 1.68
  • 1.69
Fixed in: (no value)



Subject: threads->list() in END block blocks
Currently, PAR does not work with threads. I'd love to fix it on the PAR side of things, but I'm stuck. Along the way, I think I found a bug in threads: PAR::Packer has some cleanup-code in an END{} block which must be run by the last remaining thread only. We can't do anything about detached threads, I suppose, but I thought checking that threads->list() == 1 would be sufficient for the usual case (no detached threads). So I rebuilt the situation in PAR as simple as possible. See code below. Turns out the following code gets into some locking problem or similar during the threads->list() call: #!/usr/bin/perl use strict; use warnings; use threads; sub start_thread { eval "use End;"; print "I am a thread.\n"; #END{eval"threads->list()"} } threads->create("start_thread")->join(); print "I am not a thread \n"; Whereas End.pm contains: package End; use strict; use warnings; END { print "CLEANUP!\n"; my $tid = threads->list(); if (not defined $tid or $tid == 0) { print "MAIN THREAD!\n"; } else { print "CHILD THREAD: $tid!\n"; } } 1; When I run the script, I get: I am a thread. CLEANUP! and then, it just spins. (0% CPU use, so it's not a loop.) This doesn't happen if I run the threads->list() in the main program, i.e. if I comment out the "eval"use End;"" and comment in the END{eval"threads->list()"}. Also, in reality, I'd do eval "threads->list()" in End (aka PAR) because that degrades nicely in case the user isn't even using threads. That doesn't make a difference, though. Best regards, Steffen P.S.: I tested this with a very recent 5.10-devel checkout (see below) as well as a Suse Linux 5.8.8. Both were using the newest threads (1.69) and both showed the behaviour described. I also tested the 5.8.8 with various threads releases. It seems that threads 1.57 is the first release with the locking problem. P.P.S: If you have any suggestions how to fix the original problem I'm trying to work around, I'm all ears! This is perl, v5.10.0 DEVEL33787 built for i686-linux-thread-multi (with 1 registered patch, see perl -V for more detail) Copyright 1987-2007, Larry Wall Perl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Perl 5 source kit. Complete documentation for Perl, including FAQ lists, should be found on this system using "man perl" or "perldoc perl". If you have access to the Internet, point your browser at http://www.perl.org/, the Perl Home Page. OKG72|smueller@iklx107:/userdata2/smueller/par> perl -V Summary of my perl5 (revision 5 version 10 subversion 0 patch 33787) configuration: Platform: osname=linux, osvers=2.6.5-7.257-smp, archname=i686-linux-thread-multi uname='linux iklx107 2.6.5-7.257-smp #1 smp mon may 15 14:14:14 utc 2006 i686 athlon i386 gnulinux ' config_args='-Dusedevel -Dprefix=/userdata2/smueller/bleadperl/install5.10.107 -Dusethreads -Dsiteprefix=/userdata2/smueller/bleadperl/install5.10.107' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='ccache cc -m32', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -I/usr/local/include' ccversion='', gccversion='3.3.3 (SuSE Linux)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='ccache cc -m32', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.3.3' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API Locally applied patches: MAINT33535 Built under linux Compiled at May 6 2008 13:59:36 %ENV: PERL5LIB="/users/ik3al1/smueller/local/perl5lib:/users/ik3al1/smueller/perl/lib/perl5/site_perl/5.10.0:/users/ik3al1/smueller/perl/lib/perl5/5.10.0:/users/ik3al1/smueller/perl/lib/5.10.0:/users/ik3al1/smueller/perl/lib/site_perl/5.10.0" PERLVERSION="5.10.0" @INC: /users/ik3al1/smueller/local/perl5lib /users/ik3al1/smueller/perl/lib/perl5/site_perl/5.10.0/i686-linux-thread-multi /users/ik3al1/smueller/perl/lib/perl5/site_perl/5.10.0 /users/ik3al1/smueller/perl/lib/perl5/5.10.0/i686-linux-thread-multi /users/ik3al1/smueller/perl/lib/perl5/5.10.0 /users/ik3al1/smueller/perl/lib/5.10.0/i686-linux-thread-multi /users/ik3al1/smueller/perl/lib/5.10.0 /users/ik3al1/smueller/perl/lib/site_perl/5.10.0/i686-linux-thread-multi /users/ik3al1/smueller/perl/lib/site_perl/5.10.0 /userdata2/smueller/bleadperl/install5.10.107/lib/5.10.0/i686-linux-thread-multi /userdata2/smueller/bleadperl/install5.10.107/lib/5.10.0 /userdata2/smueller/bleadperl/install5.10.107/lib/site_perl/5.10.0/i686-linux-thread-multi /userdata2/smueller/bleadperl/install5.10.107/lib/site_perl/5.10.0 ~smueller/local/perl5lib .
Subject: Re: [rt.cpan.org #35836] threads->list() in END block blocks
Date: Tue, 13 May 2008 12:00:26 -0400
To: bug-threads [...] rt.cpan.org
From: "Jerry D. Hedden" <jdhedden [...] cpan.org>
Steffen Mueller wrote: Show quoted text
> PAR::Packer has some cleanup-code in an END{} block which must be run by > the last remaining thread only. > > #!/usr/bin/perl > use strict; use warnings; > use threads; > > sub start_thread { > eval "use End;"; > print "I am a thread.\n"; > #END{eval"threads->list()"} > } > > threads->create("start_thread")->join(); > print "I am not a thread \n"; > > Whereas End.pm contains: > package End; > use strict; use warnings; > END { > print "CLEANUP!\n"; > my $tid = threads->list(); > if (not defined $tid or $tid == 0) { > print "MAIN THREAD!\n"; > } else { > print "CHILD THREAD: $tid!\n"; > } > } > 1; > > When I run the script, I get: > I am a thread. > CLEANUP! > > and then, it just spins. (0% CPU use, so it's not a loop.)
When I try this, I get: panic: MUTEX_LOCK (45) [threads.xs:1001] at End.pm line 5. END failed--call queue aborted at ./xxx.pl line 10. (in cleanup) panic: MUTEX_LOCK (45) [threads.xs:202] at ./xxx.pl line 11. In the ->join() call, the main thread holds a mutex on the thread being joined during the destruction of its interperter. While this mutex is held, the thread's END block is executed. The ->list() call in the END block then tries to obtain the same mutex. This results in the panic. Thus, running threads calls in a thread's END block will most likely fail this way. These mutex's are essential for proper operation of the threads module, and cannot be circumvented. I will document this issue for the next release of the threads module. However, it is not reliable to use ->list() to check for the number of threads. Two threads could do the same check at nearly the same time and each see 2 threads running and then both would terminate without executing the desired code. Try eliminating the END block all together, and track threads in the main code. Increment a counter on ->create() and decrement on ->join(). Then execute the "END" code after the final ->join(). Another way might be to provide threads with initialization and termination functions that track thread counts: use Thread::Semaphore; my $sema = Thread->Semaphore(); my $thread_count = 0; sub thr_init() { $sema->down(); $thread_count++; $sema->up(); } sub thr_exit() { my $thr_cnt; $sema->down(); $thr_cnt = --$thread_count; $sema->up(); if ($thr_cnt == 0) { # I'm the last thread, so do extra work } threads->exit(); } sub thr_func() { thr_init(); # Do work thr_exit(); } If you really have to have an END block, it might be safe to use the thr_exit() call in it (but I cringe at the thought).
Subject: Re: [rt.cpan.org #35836] threads->list() in END block blocks
Date: Tue, 13 May 2008 12:15:06 -0400
To: bug-threads [...] rt.cpan.org
From: "Jerry D. Hedden" <jdhedden [...] cpan.org>
Jerry D. Hedden wrote: Show quoted text
> I will document this issue for the next release of the > threads module.
I have added the following to the BUGS AND LIMITATIONS section of the threads POD which will appear in the next release: =item END blocks in threads It is possible to add L<END blocks|perlmod/"BEGIN, UNITCHECK, CHECK, INIT and END"> to threads by L<require|perlfunc/"require VERSION">'ing or L<eval|perlfunc/"eval EXPR">'ing the appropriate code. These C<END> blocks will then be executed when the thread's interpreter is destroyed (i.e., either during a C<-E<gt>join()> call, or at program termination). However, calling any L<threads> methods in such an C<END> block will most likely I<fail> (e.g., the application may hang, or genereate an error) due to mutex's that are needed to control functionality within the L<threads> module. For this reason, the use of C<END> blocks in threads is B<strongly> discouraged.
Subject: Re: [rt.cpan.org #35836] threads->list() in END block blocks
Date: Wed, 14 May 2008 11:52:30 +0200
To: bug-threads [...] rt.cpan.org
From: Steffen Mueller <wyp3rlx02 [...] sneakemail.com>
Hi Jerry, Jerry D. Hedden via RT wrote: Show quoted text
> Try eliminating the END block all together, and track > threads in the main code. Increment a counter on ->create() > and decrement on ->join(). Then execute the "END" code after > the final ->join().
That's exactly my problem. I'd happily do so, but I'm having the problem with a library: PAR is used by users unsuspecting of what kind of cleanup it has to do. They also use threads the same way. When they use PAR and threads together, PAR can't do its cleanup for the reasons described earlier. Essentially, this pushes an implementation detail of PAR out to become an obligation of the user. That's really, really bad. That being said, I understand that it's hard to impossible to fix this on the threads side of things. Unfortunately, the same applies to PAR. We have to use the END block. I know of no other way to run clean-up code on exit without user intervention. Moreover, users expect to be able to take just any arbitrary existing Perl program and run the PAR packager pp -o binary threaded-program.pl to get a working binary package. Quite often, the PAR users aren't even those who wrote the application in the first place, so there is no way we, as the PAR developers, can impose any kind of responsibility for cleanup and/or tracking of threads on the user. Show quoted text
> Another way might be to provide threads with initialization > and termination functions that track thread counts:
[...] Show quoted text
> If you really have to have an END block, it might be safe > to use the thr_exit() call in it (but I cringe at the > thought).
That's still in user-land, so no, I can't do this :( Thanks for your input and explanation. The note you put in the documentation reads well. I'll think about the issue some more, but I'm not confident I'll have any kind of revelation. Maybe I'll just add an if( threads was never use()d ) { # run cleanup } else { # don't } to the end block. Then the users using threads would sometimes have stale caches on their /tmp disk, but that's better than segmentation faults or processes locking up. Best regards, Steffen
Jerry D. Hedden wrote: Show quoted text
> I will document this issue for the next release of the > threads module.
Released