Bug #44626 for forks: using fork() breaks shared variables

Fri Mar 27 11:53:59 2009 bitcard [...] faxm0dem.org - Ticket created

Subject:

using fork() breaks shared variables

Running attached code should yield (using threads and threads::shared): -- e : 5 c : 3 a : 1 b : 2 d : 4 f : 6 a=1000 -- or similar. This works on a linux and a solaris box. Using forks and forks::shared yields on a debian linux (testing): -- e : 1 5 : c a=a -- it hangs on solaris 10u5 using cpan forks and stock perl 5.8.4

Subject:

forks

Download forks
application/octet-stream 310b

Message body not shown because it is not plain text.

Tue Apr 07 01:53:42 2009 RYBSKEJ [...] cpan.org - Taken

Tue Apr 07 01:53:52 2009 RYBSKEJ [...] cpan.org - Status changed from 'new' to 'open'

Tue Apr 07 02:56:21 2009 RYBSKEJ [...] cpan.org - Correspondence added

What are you attempting to accomplish here? Is there a specific CPAN module you are trying to use? Can your problem be solved without exiting from the parent process (main thread)? There is a fundamental difference between forks and threads in this case. Since forks.pm uses fork() to create new threads, each thread is a kernel process; however, with threads.pm, all threads reside in a single process, so fork() actually makes a copy of _all_ running threads. This means parent and child processes are running the same set of threads. Because of this different context of fork() with threads, forks.pm can not emulate this behavior. In the case of native threads.pm, fork() behavior could be considered a benefit or an issue; it does allow threads.pm to be generally compatible with process-control interface modules (like App::Daemon, which behaves similarly to the test script you referenced in this bug report); however, it can confuse/break some threaded modules (like DBIx::Threaded) which depend on a single instance of a shared resource, and where fork() will duplicate shared resources that cannot be shared used in two processes. I generally consider it risky to mix native threads with fork(), given possible side-effects and confusing/unintended exit() behaviors. Regarding forks.pm, fork() from a thread should generally behave OK, _except_ when attempting to exit() in the main thread (parent process of script). This will terminate the entire process group (all threads). Note that since forks.pm implements the same exit() behavior as threads.pm, exit() will shut down all running threads/processes, so threads- Show quoted text

>exit or 'use threads 'exit' => 'threads_only'" would be appropriate. (See http://search.cpan.org/~jdhedden/threads-1.72/threads.pm#EXITING_A_THREAD).

So, it may be feasible to add a new feature in forks.pm to allow demonization of a threaded app: termination of main thread without exiting process group. I'd imagine it would logically reassign main thread ownership to the child fork() process (auto-casting the forked process as a thread->isthread), such that ithread semantics remain consistent. But this would still not behave the same as threads.pm for reasons noted earlier, so I wonder if an alternate solution would better suit your design needs?

Tue Apr 07 08:00:01 2009 bitcard [...] faxm0dem.org - Correspondence added

On Tue Apr 07 02:56:21 2009, RYBSKEJ wrote: Show quoted text

> What are you attempting to accomplish here? Is there a specific > CPAN module you are > trying to use? Can your problem be solved without exiting from the > parent process (main > thread)?

First of all thanks a lot for taking the time to respond with so much detail. I'm not familiar enough with threads(.pm) or forks(.pm) for that matter to understand all of it though. Secondly, let me add background information. Indeed my initial code is a daemon using threads. The daemon implementation is almost exactly the same as in 'perldoc perlipc' (daemonize): 1- read configuration 2- daemonize 3- start threads The problem I have is that after (2), most of my variables initialized in (1) are empty. I tried to simulate this in attached example case.

Wed Nov 10 10:39:04 2010 bitcard [...] faxm0dem.org - Correspondence added

From:

bitcard [...] faxm0dem.org

Hi, Le Mar 07 Avr 2009 02:56:21, RYBSKEJ a écrit : Show quoted text

> So, it may be feasible to add a new feature in forks.pm to allow > demonization of a threaded > app: termination of main thread without exiting process group. I'd > imagine it would logically reassign main thread ownership to the child > fork() process (auto-casting the forked process > as a thread->isthread), such that ithread semantics remain consistent. > But this would still > not behave the same as threads.pm for reasons noted earlier, so I > wonder if an alternate > solution would better suit your design needs?

I'm still struggling with this issue. What would be the best way to implement a daemon using forks.pm ?

Sat Nov 13 06:41:00 2010 RYBSKEJ [...] cpan.org - Correspondence added

I generally recommend shell-based tools to manage process demonization, whether using forks.pm or not. These avoid any possible side effects or compatibility issues. Some ideas: - supervise (part of the daemontools package): http://cr.yp.to/daemontools.html - panctl: http://search.cpan.org/search?query=panctl&mode=all - Roll your own shell script If writing your own script, I recommend reviewing the 'apachectl' script included with the Apache httpd distribution. This is a nice example of a complete shell daemon script manager, SysV command style. For managed log rotation, consider some the following: - multilog (http://cr.yp.to/daemontools/multilog.html): Runs in-line with your script, or piped output via Perl open(). No signal handling necessary. - logrotate: Runs externally from your script. You must handle a custom HUP signal. Can handle any log file - Log4Perl with RollingFileAppender or SysLogAppender: Can handle any log file. Most configurable Perl solution Personally, I always write my own daemon-control scripts, frequently in combination with multilog. On the original bug topic, I had evaluated fork()/exit from forks.pm main thread and not terminating the thread group. I was dissatisfied with internal side effects with the supervisor thread (e.g. it becomes an orphan process, with respect to the script). So I'd rather not add this feature to the forks.pm core, as I feel it would cause more instability and cleanup issues than simply using a wrapper daemon script or utility. On Wed Nov 10 10:39:04 2010, faxm0dem wrote: Show quoted text

> Hi, > > Le Mar 07 Avr 2009 02:56:21, RYBSKEJ a écrit :

> > So, it may be feasible to add a new feature in forks.pm to allow > > demonization of a threaded > > app: termination of main thread without exiting process group. I'd > > imagine it would logically reassign main thread ownership to the child > > fork() process (auto-casting the forked process > > as a thread->isthread), such that ithread semantics remain consistent. > > But this would still > > not behave the same as threads.pm for reasons noted earlier, so I > > wonder if an alternate > > solution would better suit your design needs?

> > I'm still struggling with this issue. > What would be the best way to implement a daemon using forks.pm ?

Mon Nov 15 04:42:06 2010 bitcard [...] faxm0dem.org - Correspondence added

From:

bitcard [...] faxm0dem.org

Thank you for looking into this so thoroughly and having considered changing the code. Before wrapping up my script into shell I followed an other idea that was to load either threads or forks at runtime, so I could run the daemonize code *before* forks (or threads that is). So now I basically have: -- &daemonize if $opt{daemonize}; if ($opt{threads}) { eval "use threads"; } else { eval "use forks"; } &share_config; [...] -- That seems sane to me, as the daemonized child would become the master thread, instead of its parent. Now I run into something different: the share() doesn't seem to work. Moreover, the routine seems to have a different prototype (also happens when using 'require' instead of 'use'): in one case "share()" seems to accept options as documented: "Type of arg 1 to threads::shared::share must be one of [$@%] (not single ref constructor)" And in the other: "Argument to share needs to be passed as ref at -e line 1." or "Usage: threads::shared::share(myref) at -e line 1." Also see e.g. http://www.mail-archive.com/perl-ithreads@perl.org/msg00628.html I am obviously missing something here.

Mon Nov 15 09:27:32 2010 bitcard [...] faxm0dem.org - Correspondence added

From:

bitcard [...] faxm0dem.org

Apart from this prototype thing, I successfully worked around the issue, and it works nicely forking before the eval blocks loading the module. Thanks for your help

Wed Jun 25 22:27:21 2014 RYBSKEJ [...] cpan.org - Correspondence added

On Mon Nov 15 09:27:32 2010, faxm0dem wrote: Show quoted text

> Apart from this prototype thing, I successfully worked around the issue, > and it works nicely forking before the eval blocks loading the module. > > Thanks for your help

I will mark this resolved for now, as you found a satisfactory resolution. Please feel free to open a new ticket if you feel further research is warranted.

Wed Jun 25 22:27:22 2014 RYBSKEJ [...] cpan.org - Status changed from 'open' to 'resolved'