Skip Menu |

This queue is for tickets about the IPC-Exe CPAN distribution.

Report information
The Basics
Id: 45413
Status: resolved
Priority: 0/
Queue: IPC-Exe

People
Owner: glai [...] cpan.org
Requestors: sergstesh [...] yahoo.com
Cc:
AdminCc:

Bug Information
Severity: Critical
Broken in: 1.005
Fixed in: 1.006



Subject: A chain of trivial scripts is stuck under Win2k, works under Linux
I have created a main program (see the attached 'try_ipc_win2k_bug.pl' file) which call three other scripts (see the attached 'left.pl', 'center.pl', 'right.pl'). In order to run the suite the following needs to be done: 1) choose whatever work directory; 2) cd to the above directory; 3) download all the above file into current directory; 2) run perl -w try_ipc_win2k_bug.pl . Output under Linux looks like this, and it is expected: " (0) $$=26808 at /mnt/sdb8/sergei/QEMU/portable_perl/try_ipc_win2k_bug.pl line 9. (1) $$=26809 at /mnt/sdb8/sergei/QEMU/portable_perl/try_ipc_win2k_bug.pl line 18. left.pl: STTDERR: @ARGV=1 at left.pl line 5. (4) $$=26808 at /mnt/sdb8/sergei/QEMU/portable_perl/try_ipc_win2k_bug.pl line 39. Show quoted text
last_pipe> /home/sergei/PERL-5.10.0/bin/perl -w right.pl
left.pl: STTDERR: @ARGV=2 at left.pl line 5. +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=1 $rand=0343142 +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=2 $rand=0198413 left.pl: STTDERR: @ARGV=3 at left.pl line 5. +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=3 $rand=0557620 left.pl: STTDERR: @ARGV=4 at left.pl line 5. +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=4 $rand=0564594 left.pl: STTDERR: @ARGV=5 at left.pl line 5. +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=5 $rand=0665700 left.pl: STTDERR: @ARGV=6 at left.pl line 5. +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=6 $rand=0371080 left.pl: STTDERR: @ARGV=7 at left.pl line 5. +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=7 $rand=0327316 left.pl: STTDERR: @ARGV=8 at left.pl line 5. +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=8 $rand=0668209 left.pl: STTDERR: @ARGV=9 at left.pl line 5. +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=9 $rand=0049729 left.pl: STTDERR: @ARGV=10 at left.pl line 5. +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=10 $rand=0781647 @pids=26809 26811 26812 0 at /mnt/sdb8/sergei/QEMU/portable_perl/try_ipc_win2k_bug.pl line 62, <STDIN> line 10. ". When I run the same setup under Windows 2000 (under QEMU to be exact), the command line and its screen output is like this: " C:\perl_plus_player>C:\Perl\bin\perl.exe -w try_ipc_win2k_bug.pl (0) $$=580 at try_ipc_win2k_bug.pl line 9. (1) $$=-860 at try_ipc_win2k_bug.pl line 18. left.pl: STTDERR: @ARGV=1 at left.pl line 5. left.pl: STTDERR: @ARGV=2 at left.pl line 5. left.pl: STTDERR: @ARGV=3 at left.pl line 5. left.pl: STTDERR: @ARGV=4 at left.pl line 5. left.pl: STTDERR: @ARGV=5 at left.pl line 5. left.pl: STTDERR: @ARGV=6 at left.pl line 5. left.pl: STTDERR: @ARGV=7 at left.pl line 5. left.pl: STTDERR: @ARGV=8 at left.pl line 5. ", after that the script is stuck. Actually, the script is not always stuck. If in 'try_ipc_win2k_bug.pl' file one changes this line: 19 for my $ctr(1 .. 10) to become for my $ctr(1 .. 2) , the script works: " C:\perl_plus_player>C:\Perl\bin\perl.exe -w try_ipc_win2k_bug.pl (0) $$=1024 at try_ipc_win2k_bug.pl line 9. (1) $$=-968 at try_ipc_win2k_bug.pl line 18. left.pl: STTDERR: @ARGV=1 at left.pl line 5. left.pl: STTDERR: @ARGV=2 at left.pl line 5. (4) $$=1024 at try_ipc_win2k_bug.pl line 39. Show quoted text
last_pipe> C:\Perl\bin\perl.exe -w right.pl
+++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=1 $rand=0089996 +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: STDOUT: @ARGV=2 $rand=0420715 @pids=-968 -692 -704 0 at try_ipc_win2k_bug.pl line 62, <STDIN> line 2. C:\perl_plus_player> ". What else ? I've noticed that under Win2k if Perl 'system("dir");' is called inside a thread, the whole thing gets stuck - I do not have a ready code example at hand. Also, the following bug reports: http://bugs.activestate.com/show_bug.cgi?id=82846 http://bugs.activestate.com/show_bug.cgi?id=82870 might be related. The failure occurs under the latest ActiveState perl-5.10.0; according to my tries with other Perl modules and scripts of mine under win2k the flakiness doesn't really depend on Perl distro - say, Strawberry Perl is equally good/bad.
Subject: right.pl
#!/home/sergei/PERL-5.10.0/bin/perl -w $| = 1; use strict; while(defined(my $line = <STDIN>)) { print "+++processed by right.pl+++: $line" } exit(0);
Subject: left.pl
#!/home/sergei/PERL-5.10.0/bin/perl -w use strict; warn "left.pl: STTDERR: \@ARGV=@ARGV"; my $rand = sprintf('%07u', int(rand(1000000))); print STDOUT "left.pl: STDOUT: \@ARGV=@ARGV \$rand=$rand\n"; exit(0);
Subject: try_ipc_win2k_bug.pl
#!/home/sergei/PERL-5.10.0/bin/perl -w use IPC::Exe qw(exe bg); use strict; $| = 1; warn "(0) \$\$=$$"; my $program = '/bin/ls'; my @pids = &{ exe ( sub { $| = 1; warn "(1) \$\$=$$"; for my $ctr(1 .. 10) { #print STDOUT "$ctr\n"; &{exe $^X, '-w', 'left.pl', $ctr}; } }, # 'perl' receives STDOUT of $program via STDIN exe ( $^X, '-w', 'center.pl', exe ( $^X, '-w', 'right.pl', exe ( sub { warn "(4) \$\$=$$"; # find out command of previous pipe process # if @_ is empty list, previous process was a [perlsub] my ($prog, @args) = @_; print STDERR "last_pipe> $prog @args\n"; # output: "last_pipe> sort -n" # print sorted, 'perl' filtered, output of $program print while <STDIN>; # find out exit value of previous 'sort' pipe process close($IPC::Exe::PIPE); warn("Bad exit for: @_\n") if $?; return $?; }, ) ) ) ) }; warn "\@pids=@pids"
Subject: center.pl
#!/home/sergei/PERL-5.10.0/bin/perl -w $| = 1; use strict; while(defined(my $line = <STDIN>)) { print "<<<processed by center.pl>>>: $line"; } exit(0);
Thank you so much for exercising IPC::Exe, Sergei. I have been able to reproduce the exact problem of what you're seeing on my side. I think I understand the problem now. It comes down to fundamentals of pipes. You see, writing and reading from a pipe causes blocking. Since we only have threads in Windows, once it blocks, it blocks forever. IPC::Exe exists in the thread and needs to carry on. What this means is you cannot have a writer and a reader in the same thread (unless you can poll the reader/writer continuously). This greatly reduces what &PREEXEC can do. The cleanest solution is to have the pipe writer and reader in two separate processes. Since IPC::Exe takes care of pipe readers in the current process, you have to exec the pipe writer into another process. That means, if you want a &PREEXEC like your example below, you have to exec a 'perl' process that does the same thing, from scratch. You have given me a lot to think about, and have helped me expose many bugs. The next release IPC::Exe 1.006 will be a MAJOR update. Thanks again. -- Gerald On Fri Apr 24 23:50:13 2009, sergstesh wrote: Show quoted text
> I have created a main program (see the attached 'try_ipc_win2k_bug.pl' > file) which call three other scripts (see the attached 'left.pl', > 'center.pl', 'right.pl'). > > In order to run the suite the following needs to be done: > > 1) choose whatever work directory; > 2) cd to the above directory; > 3) download all the above file into current directory; > 2) run > > perl -w try_ipc_win2k_bug.pl > . > > Output under Linux looks like this, and it is expected: > > " > (0) $$=26808 at /mnt/sdb8/sergei/QEMU/portable_perl/try_ipc_win2k_bug.pl > line 9. > (1) $$=26809 at /mnt/sdb8/sergei/QEMU/portable_perl/try_ipc_win2k_bug.pl > line 18. > left.pl: STTDERR: @ARGV=1 at left.pl line 5. > (4) $$=26808 at /mnt/sdb8/sergei/QEMU/portable_perl/try_ipc_win2k_bug.pl > line 39.
> last_pipe> /home/sergei/PERL-5.10.0/bin/perl -w right.pl
> left.pl: STTDERR: @ARGV=2 at left.pl line 5. > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=1 $rand=0343142 > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=2 $rand=0198413 > left.pl: STTDERR: @ARGV=3 at left.pl line 5. > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=3 $rand=0557620 > left.pl: STTDERR: @ARGV=4 at left.pl line 5. > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=4 $rand=0564594 > left.pl: STTDERR: @ARGV=5 at left.pl line 5. > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=5 $rand=0665700 > left.pl: STTDERR: @ARGV=6 at left.pl line 5. > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=6 $rand=0371080 > left.pl: STTDERR: @ARGV=7 at left.pl line 5. > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=7 $rand=0327316 > left.pl: STTDERR: @ARGV=8 at left.pl line 5. > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=8 $rand=0668209 > left.pl: STTDERR: @ARGV=9 at left.pl line 5. > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=9 $rand=0049729 > left.pl: STTDERR: @ARGV=10 at left.pl line 5. > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=10 $rand=0781647 > @pids=26809 26811 26812 0 at > /mnt/sdb8/sergei/QEMU/portable_perl/try_ipc_win2k_bug.pl line 62, > <STDIN> line 10. > ". > > When I run the same setup under Windows 2000 (under QEMU to be exact), > the command line and its screen output is like this: > > " > C:\perl_plus_player>C:\Perl\bin\perl.exe -w try_ipc_win2k_bug.pl > (0) $$=580 at try_ipc_win2k_bug.pl line 9. > (1) $$=-860 at try_ipc_win2k_bug.pl line 18. > left.pl: STTDERR: @ARGV=1 at left.pl line 5. > left.pl: STTDERR: @ARGV=2 at left.pl line 5. > left.pl: STTDERR: @ARGV=3 at left.pl line 5. > left.pl: STTDERR: @ARGV=4 at left.pl line 5. > left.pl: STTDERR: @ARGV=5 at left.pl line 5. > left.pl: STTDERR: @ARGV=6 at left.pl line 5. > left.pl: STTDERR: @ARGV=7 at left.pl line 5. > left.pl: STTDERR: @ARGV=8 at left.pl line 5. > ", > > after that the script is stuck. > > Actually, the script is not always stuck. If in 'try_ipc_win2k_bug.pl' > file one changes this line: > > 19 for my $ctr(1 .. 10) > > to become > > for my $ctr(1 .. 2) > > , the script works: > > " > C:\perl_plus_player>C:\Perl\bin\perl.exe -w try_ipc_win2k_bug.pl > (0) $$=1024 at try_ipc_win2k_bug.pl line 9. > (1) $$=-968 at try_ipc_win2k_bug.pl line 18. > left.pl: STTDERR: @ARGV=1 at left.pl line 5. > left.pl: STTDERR: @ARGV=2 at left.pl line 5. > (4) $$=1024 at try_ipc_win2k_bug.pl line 39.
> last_pipe> C:\Perl\bin\perl.exe -w right.pl
> +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=1 $rand=0089996 > +++processed by right.pl+++: <<<processed by center.pl>>>: left.pl: > STDOUT: @ARGV=2 $rand=0420715 > @pids=-968 -692 -704 0 at try_ipc_win2k_bug.pl line 62, <STDIN> line 2. > > C:\perl_plus_player> > ". > > What else ? I've noticed that under Win2k if Perl 'system("dir");' is > called inside a thread, the whole thing gets stuck - I do not have a > ready code example at hand. > > Also, the following bug reports: > > http://bugs.activestate.com/show_bug.cgi?id=82846 > http://bugs.activestate.com/show_bug.cgi?id=82870 > > might be related. > > The failure occurs under the latest ActiveState perl-5.10.0; according > to my tries with other Perl modules and scripts of mine under win2k the > flakiness doesn't really depend on Perl distro - say, Strawberry Perl is > equally good/bad.
On Sun Apr 26 15:14:54 2009, GLAI wrote: Show quoted text
> Thank you so much for exercising IPC::Exe, Sergei. I have been able to > reproduce the exact problem of what you're seeing on my side. > > I think I understand the problem now. It comes down to fundamentals of > pipes. You see, writing and reading from a pipe causes blocking. Since > we only have threads in Windows, once it blocks, it blocks forever. > IPC::Exe exists in the thread and needs to carry on. What this means is > you cannot have a writer and a reader in the same thread (unless you can > poll the reader/writer continuously). This greatly reduces what &PREEXEC > can do. > > The cleanest solution is to have the pipe writer and reader in two > separate processes. Since IPC::Exe takes care of pipe readers in the > current process, you have to exec the pipe writer into another process. > That means, if you want a &PREEXEC like your example below, you have to > exec a 'perl' process that does the same thing, from scratch. > > You have given me a lot to think about, and have helped me expose many > bugs. The next release IPC::Exe 1.006 will be a MAJOR update. > > Thanks again. > -- > Gerald >
Gerald, thanks for prompt reaction to my request. After reading your reply I do not quite understand how it explains the fact that if I shorten the generating loop the whole thing is not stuck under Win2k. Let me explain what the attached scripts are meant to do. First and simplistically try_ipc_win2k_bug.pl is supposed to do moral equivalent of system("left.pl | center.pl | right.pl"); Now, to be more precise, 'left.pl' is not supposed to be called just once, but in a loop inside 'try_ipc_win2k_bug.pl': 19 for my $ctr(1 .. 10) 20 { 21 #print STDOUT "$ctr\n"; 22 &{exe $^X, '-w', 'left.pl', $ctr}; 23 } . So, as I said, if I just replace 10 in "for my $ctr(1 .. 10)" with 2, the whole thing is not stuck under Win2k. I understand there are only threads under Windows, but I do not yet see how changing number of iterations in the generating loop changes the fact of (non)blocking. Had the whole thing behaved in a manned according to the case with 2 iterations, i.e. had it printed first something and then stopped, I would have understood. Did you read http://bugs.activestate.com/show_bug.cgi?id=82846 http://bugs.activestate.com/show_bug.cgi?id=82870 ? One of the points mentioned there is that if I just have one father <-> son pair, the example with pipes works - still with threads and not real processes. If, however, I transform the example to father <-> son <-> grandson, it stops working. Maybe the problem is not in your module, but in Perl core + threads, but my knowledge is not enough to really prove it. I believe that with my examples Win32 Perl 'fork' implementation should be equivalent to real 'fork' but maybe I misunderstood the manual. Thanks, Sergei.
On Sun Apr 26 16:35:53 2009, sergstesh wrote: Show quoted text
> On Sun Apr 26 15:14:54 2009, GLAI wrote:
> > Thank you so much for exercising IPC::Exe, Sergei. I have been able to > > reproduce the exact problem of what you're seeing on my side. > > > > I think I understand the problem now. It comes down to fundamentals of > > pipes. You see, writing and reading from a pipe causes blocking. Since > > we only have threads in Windows, once it blocks, it blocks forever. > > IPC::Exe exists in the thread and needs to carry on. What this means is > > you cannot have a writer and a reader in the same thread (unless you can > > poll the reader/writer continuously). This greatly reduces what &PREEXEC > > can do. > > > > The cleanest solution is to have the pipe writer and reader in two > > separate processes. Since IPC::Exe takes care of pipe readers in the > > current process, you have to exec the pipe writer into another process. > > That means, if you want a &PREEXEC like your example below, you have to > > exec a 'perl' process that does the same thing, from scratch. > > > > You have given me a lot to think about, and have helped me expose many > > bugs. The next release IPC::Exe 1.006 will be a MAJOR update. > > > > Thanks again. > > -- > > Gerald > >
> > Gerald, thanks for prompt reaction to my request. > > After reading your reply I do not quite understand how it explains the > fact that if I shorten the generating loop the whole thing is not stuck > under Win2k. > > Let me explain what the attached scripts are meant to do. > > First and simplistically try_ipc_win2k_bug.pl is supposed to do moral > equivalent of > > system("left.pl | center.pl | right.pl"); > > Now, to be more precise, 'left.pl' is not supposed to be called just > once, but in a loop inside 'try_ipc_win2k_bug.pl': > > 19 for my $ctr(1 .. 10) > 20 { > 21 #print STDOUT "$ctr\n"; > 22 &{exe $^X, '-w', 'left.pl', $ctr}; > 23 } > . > > So, as I said, if I just replace 10 in "for my $ctr(1 .. 10)" with 2, > the whole thing is not stuck under Win2k. > > I understand there are only threads under Windows, but I do not yet see > how changing number of iterations in the generating loop changes the > fact of (non)blocking. > > Had the whole thing behaved in a manned according to the case with 2 > iterations, i.e. had it printed first something and then stopped, I > would have understood. > > Did you read > > http://bugs.activestate.com/show_bug.cgi?id=82846 > http://bugs.activestate.com/show_bug.cgi?id=82870 > > ? > > One of the points mentioned there is that if I just have one father <-> > son pair, the example with pipes works - still with threads and not real > processes. > > If, however, I transform the example to father <-> son <-> grandson, it > stops working. > > Maybe the problem is not in your module, but in Perl core + threads, but > my knowledge is not enough to really prove it. > > I believe that with my examples Win32 Perl 'fork' implementation should > be equivalent to real 'fork' but maybe I misunderstood the manual. > > Thanks, > Sergei.
Sergei, Yes, I read your bug reports. The reason why shortening the loop avoids the deadlock is because then you won't be printing out _too much_ to the pipe. It is blocking because the pipe is full, and the writer is waiting till the reader reads off some information from the pipe before continuing. However, the writer is not in its own separate process as in a real fork() - when it blocks, it prevents &READER (same process, separate thread) from continuing for subsequent exe's. So, is the reader (center.pl) ready to receive data? Nope. Unfortunately, center.pl was not even spawned yet because IPC::Exe was blocked by print(). Please allow me time to address this. The documentation will be updated to explain how to use IPC::Exe effectively in Win32. Thanks for your patience, Gerald
Sergei, I've made improvements to v1.006 and updated the POD. Please read the CAVEAT section. In your specific case of try_ipc_win2k_bug.pl, you hit a limitation of threads. If you moved the for-loop inside left.pl and do exe $^X, '-w', 'left.pl', $ctr, instead of exe sub { $| = 1; warn "(1) \$\$=$$"; for my $ctr(1 .. 10) { &{exe $^X, '-w', 'left.pl', $ctr}; } }, it will work. Let me know if you have further questions. Thanks again. -- Gerald