Skip Menu |

This queue is for tickets about the MailTools CPAN distribution.

Report information
The Basics
Id: 99153
Status: resolved
Priority: 0/
Queue: MailTools

People
Owner: Nobody in particular
Requestors: kissg [...] niif.hu
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in:
  • 2.09
  • 2.13
Fixed in: (no value)



Subject: Race condition in Mail::Field::import()
In multithreaded environment Mail::Field::import() can call _require_dir() with $dir='/' causing endless loop on Linux machines because it wants to crawl /proc too. Debugging OpenCA we found that when import() iterates over %INC it does not check the value. However in 20-30% of cases the key ($f) valid but the value ($INC{$f}) is empty yet. $ perl -V Summary of my perl5 (revision 5 version 18 subversion 2) configuration: Platform: osname=linux, osvers=3.6.10-8.fc18.armv7hl.highbank, archname=arm-linux-thread-multi uname='linux cal-7-5 3.6.10-8.fc18.armv7hl.highbank #1 smp tue jan 29 14:01:38 est 2013 armv7l armv7l armv7l gnulinux ' [...] $ uname -a Linux <hostname_here> 3.12.26-1.20140808git4ab8abb.rpfr20.armv6hl.bcm2708 #1 PREEMPT Fri Aug 8 17:13:15 EDT 2014 armv6l armv6l armv6l GNU/Linux My quick and dirty hack: foreach my $f (keys %INC) { next if $f !~ /^Mail(\W)Field\W/i; $dir_sep = $1; # kissg@niif.hu 2014-09-26 # There is a race condition here due to multithreading. # $f may have a value while $INC{$f} is empty yet. # Actually we should run this cycle for "values %INC". But a one-liner hack: return unless $INC{$f} =~ /Mail\W+Field/; $dir = ($INC{$f} =~ /(.*Mail\W+Field)/i)[0] . $dir_sep; last; }
On Fri Sep 26 03:33:34 2014, kissg@niif.hu wrote: Show quoted text
> Debugging OpenCA we found that when import() iterates over %INC it > does not check the value. However in 20-30% of cases the key ($f) > valid but the value ($INC{$f}) is empty yet.
Multi-threading in Perl is a very bad idea. I have no idea why this should happen. Who is removing paths from %INC? There may be ugly internal which involve %INC, but still I cannot find any why this could be a race condition: the keys are *always* known before you can run keys() on it. $f is always set later thanthe key. Probably the problem is best solved by loading Mail::Field before you start the threads.
From: autumn [...] autumnlansing.com
I believe I'm experiencing this very issue after upgrading MailTools to 2.13. Starting or restarting the Catalyst development server results in race conditions about 80% of the time, and straces show it beginning when either Mail::Field or MIME::Field::ConTraEnc, which is a subclass of Mail::Field, are being called. This race condition isn't present in 2.12 though, and downgrading fixes the problem. The relevant part of my straces follows. Before these lines, everything runs normally. ********* openat(AT_FDCWD, "/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 8 getdents(8, /* 25 entries */, 32768) = 648 getdents(8, /* 0 entries */, 32768) = 0 stat("//boot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 openat(AT_FDCWD, "//boot", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 9 getdents(9, /* 35 entries */, 32768) = 1592 getdents(9, /* 0 entries */, 32768) = 0 stat("//boot/sysctl", 0x786e00) = -1 ENOENT (No such file or directory) stat("/home/autumn/Projects/Loom/script/../lib/Mail/Field/boot/sysctl.pmc", 0x7fff2aa6b0f0) = -1 ENOENT (No such file or directory) stat("/home/autumn/Projects/Loom/script/../lib/Mail/Field/boot/sysctl.pm", 0x7fff2aa6b030) = -1 ENOENT (No such file or directory) stat("/home/autumn/perl5/lib/perl5/x86_64-linux/Mail/Field/boot/sysctl.pmc", 0x7fff2aa6b0f0) = -1 ENOENT (No such file or directory) stat("/home/autumn/perl5/lib/perl5/x86_64-linux/Mail/Field/boot/sysctl.pm", 0x7fff2aa6b030) = -1 ENOENT (No such file or directory) ********* From that point forward it keeps trying to stat those same two paths over and over again, eventually tacking on extra "boot" directories and finally trying to access all sorts of files in my grub directory with such wonderful paths as: stat("/home/autumn/perl5/lib/perl5/Mail/Field/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/grub2/locale/fr.pm", 0x7fff2aa6b030) = -1 ENOENT (No such file or directory) and stat("//boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/boot/grub2/themes/openSUSE/terminal_box_e", 0x786e00) = -1 ENOENT (No such file or directory) I'm running openSUSE Linux, 64-bit (x86_64), with the 3.7.10 kernel. This happens under both perl-5.20.1 using perlbrew and my system perl, which is 5.16.2.
Subject: Re: [rt.cpan.org #99153] Race condition in Mail::Field::import()
Date: Thu, 13 Nov 2014 08:20:49 +0100
To: "autumn [...] autumnlansing.com via RT" <bug-MailTools [...] rt.cpan.org>
From: Mark Overmeer <solutions [...] overmeer.net>
* autumn@autumnlansing.com via RT (bug-MailTools@rt.cpan.org) [141113 01:21]: Show quoted text
> Queue: MailTools > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=99153 > > > stat("//boot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
Where is this "//boot" coming from? I think only one '/' get's stripped each time, so we loop in the root directory. -- Regards, MarkOv ------------------------------------------------------------------------ drs Mark A.C.J. Overmeer MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
From: autumn [...] autumnlansing.com
I'm not sure where it's getting the "//boot" from. In an strace of the server using the previous version of MailTools, "boot" doesn't turn up anywhere in the output file. If there's anything I can do to delve deeper into it on my end, please let me know.
Subject: Re: [rt.cpan.org #99153] Race condition in Mail::Field::import()
Date: Sat, 15 Nov 2014 21:50:31 +0100
To: "autumn [...] autumnlansing.com via RT" <bug-MailTools [...] rt.cpan.org>
From: Mark Overmeer <mark [...] overmeer.net>
* autumn@autumnlansing.com via RT (bug-MailTools@rt.cpan.org) [141115 19:01]: Show quoted text
> Queue: MailTools > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=99153 > > > I'm not sure where it's getting the "//boot" from. In an strace of the > server using the previous version of MailTools, "boot" doesn't turn up > anywhere in the output file.
The '//boot' must come from your code. It may start as '/'. You haven't shown me your code, so I cannot help you debug. -- Regards, MarkOv
Subject: [rt.cpan.org #99153]
Date: Mon, 17 Nov 2014 16:28:42 +0100
To: bug-MailTools [...] rt.cpan.org
From: Arne Becker <Arne_Becker [...] genua.de>
Hi. We've seen similar behaviour and I have a fix. In Mail::Field::_require_dir, a recursive "require" of the submodules is attempted. If the "require" fails, we end up with an entry in %INC like this: $INC{'Mail/Field/Date.pm'} => undef. Since there is no error checking after the eval "require ...", we just continue. No problem the first time around. But if another module does "use Mail::Field" (or MIME::Parser), sub import gets called again. And, since perl 5.18, sometimes we now find the broken %INC entry first when searching for a package matching /^Mail(\W)Field\W/i. In this case, we continue with the undefined path. _require_dir is attempted with '/' as path. We iterate through the whole file system, trying to require every file in every directory matching /^([\w\-]+)/ (with added prefix Mail::Field). When there's a recursive path, like /proc, it goes into infinite recursion. I think, this is what happens with multi-threading: Thread 1 is still working on the BEGIN{} sections, i.e. has started "require", but not finished yet. Then thread 2 comes along with its use statement and sees the still empty path, which only gets set when thread 1 finishes. Fixes: 1. In Mail::Field, sub _require_dir, include the error check: } else { $p =~ s/-/_/go; print STDERR "PackagePath is a file, requiring: ${class}::$p\n"; eval "require ${class}::$p"; die "$@" if $@; } Then the process stops right there and you get an error message that actually points to the problem. This does not fix the multi-threading problem. See 3. 2. Ensure that the modules for Date::Format and Date::Parse are there and can be "required". I think the dependency on Date::Parse may be missing, but there is a dependency in Date::Format, which is in the same module TimeDate, so maybe this is correct. 3. For the multi-threading problem, maybe it's enough to add this to sub import: foreach my $f (keys %INC) { next unless defined $INC{$f}; next if $f !~ /^Mail(\W)Field\W/i; $dir_sep = $1; $dir = ($INC{$f} =~ /(.*Mail\W+Field)/i)[0] . $dir_sep; although the correct fix seems to be to make sub import synchronized. I think, it would also be a good idea to store "I did the require of my sub modules" in a package global variable in Mail::Field. Doing it on every call of import seems wrong. Alternatively, move that part out of import and into a separate BEGIN{} block. I don't think this behaviour was introduced with 2.13, it seems it was there before. Due to the hash randomization changes in perl 5.18 it does not always trigger. - Arne
Subject: Re: [rt.cpan.org #99153]
Date: Mon, 17 Nov 2014 21:55:05 +0100
To: Arne Becker via RT <bug-MailTools [...] rt.cpan.org>
From: Mark Overmeer <mark [...] overmeer.net>
* Arne Becker via RT (bug-MailTools@rt.cpan.org) [141117 15:29]: Show quoted text
> If the "require" fails, we end up with an entry in %INC like this: > $INC{'Mail/Field/Date.pm'} => undef.
Good deduction. I understand your explanation. Show quoted text
> Since there is no error checking after the eval "require ...", we just > continue.
We cannot add error checking, because many modules may accidentally try to compile all kinds of weird (non Perl) files. We cannot break existing installations. Show quoted text
> But if another module does "use Mail::Field" (or MIME::Parser), sub > import gets called again.
Maybe even a recursive require. Show quoted text
> I think, this is what happens with multi-threading: Thread 1 is still > working on the BEGIN{} sections, i.e. has started "require", but not > finished yet. Then thread 2 comes along with its use statement and sees > the still empty path, which only gets set when thread 1 finishes.
I hate Perl threads ;-) Show quoted text
> Fixes: > 1. > In Mail::Field, sub _require_dir, include the error check:
Not possible. Sorry, the code is much older than me ;-) Show quoted text
> 2. > Ensure that the modules for Date::Format and Date::Parse are there and > can be "required". > I think the dependency on Date::Parse may be missing, but there is a > dependency in Date::Format, which is in the same module TimeDate, so > maybe this is correct.
Both dependencies are in Makefile.PL If some (Linux) destribution thinks it's optional, than please file a bug-report at them. Show quoted text
> 3. For the multi-threading problem, maybe it's enough to add this to sub > import: > > foreach my $f (keys %INC) > { next unless defined $INC{$f}; > next if $f !~ /^Mail(\W)Field\W/i; > $dir_sep = $1; > $dir = ($INC{$f} =~ /(.*Mail\W+Field)/i)[0] . $dir_sep; > > although the correct fix seems to be to make sub import synchronized.
Adding that defined $INC{$f} seems to be the best solution, yes. What do you think... is the last line equivalent to ($dir = $INC{$f}) =~ s/(Mail\W+Field).*/$1$dir_sep/; Show quoted text
> I think, it would also be a good idea to store "I did the require of my > sub modules" in a package global variable in Mail::Field. Doing it on > every call of import seems wrong. Alternatively, move that part out of > import and into a separate BEGIN{} block.
Those things in BEGIN often cause problems with mod_perl. The 'tried all' flag may cause problems with packages which are added later, and add to @INC. MailTools is one of the oldest modules, written in the time (and style) of very old Perl... where everyone was using 'do some.pl' and so on. So easy to break... If you want good email modules, use Mail::Box or the Email::* suite. Show quoted text
> I don't think this behaviour was introduced with 2.13, it seems it was > there before. Due to the hash randomization changes in perl 5.18 it does > not always trigger.
Agree. -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
Subject: [rt.cpan.org #99153]
Date: Tue, 18 Nov 2014 11:05:22 +0100
To: bug-MailTools [...] rt.cpan.org
From: Arne Becker <Arne_Becker [...] genua.de>
Show quoted text
> We cannot add error checking, because many modules may accidentally try to compile all kinds of weird (non Perl) files. We cannot break existing installations.
Then please at least add a warning: warn($@) if $@; Without this, bug tracking is unnecessarily hard. I hope the people running broken existing installations will at least be interested in warnings for "require" errors. Show quoted text
> What do you think... is the last line equivalent to ($dir = $INC{$f}) =~ s/(Mail\W+Field).*/$1$dir_sep/;
Yes. I have to say that I don't understand why there is a loop here: foreach my $f (keys %INC) Why not just use something like: $path = File::Spec->catdir("Mail", "Field.pm"); return unless defined $INC{$path}; ($dir = $INC{$path}) =~ s/(\Q$path\E).*/$1$dir_sep/; Searching for anything including Mail/Field seems so strange. I also think that the require must have succeeded for our module, when we are at the point where we run "import". Show quoted text
> If you want good email modules, use Mail::Box or the Email::* suite.
I agree, but as long as MIME::Parser uses Mail::Field, the problems remain... Thanks for your time, Arne
Subject: Re: [rt.cpan.org #99153]
Date: Tue, 18 Nov 2014 11:37:01 +0100
To: Arne Becker via RT <bug-MailTools [...] rt.cpan.org>
From: Mark Overmeer <solutions [...] overmeer.net>
* Arne Becker via RT (bug-MailTools@rt.cpan.org) [141118 10:05]: Show quoted text
> Queue: MailTools > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=99153 > >
> > We cannot add error checking, because many modules may accidentally
> try to compile all kinds of weird (non Perl) files. We cannot break > existing installations. > > Then please at least add a warning: > warn($@) if $@; > Without this, bug tracking is unnecessarily hard.
I am sure that this will lead to bug-reports and new releases. But I will make this change and accept the consequences. Show quoted text
> I have to say that I don't understand why there is a loop here: > foreach my $f (keys %INC)
My guess is that the plan is that there is public/standard directory with Mail/Field/* and additional directories which contain extensions. If you load one definition, you get them all. I would never design it this way. But back in the days... At least someone has used it :-( Show quoted text
> > If you want good email modules, use Mail::Box or the Email::* suite.
> I agree, but as long as MIME::Parser uses Mail::Field, the problems > remain...
I would not use MIME::* either -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
RT-Send-CC: autumn [...] autumnlansing.com, Arne_Becker [...] genua.de
Fix released as 2.14
Subject: Re: [rt.cpan.org #99153] Race condition in Mail::Field::import()
Date: Mon, 24 Nov 2014 09:44:00 +0100
To: bug-MailTools [...] rt.cpan.org
From: Arne Becker <Arne_Becker [...] genua.de>
On 11/21/14 17:18, Mark Overmeer via RT wrote: Show quoted text
> <URL: https://rt.cpan.org/Ticket/Display.html?id=99153 > > > Fix released as 2.14 >
yay, thanks! :-) - Arne -- genua Gesellschaft fuer Netzwerk- und Unix-Administration mbH Domagkstrasse 7, 85551 Kirchheim bei Muenchen tel +49 89 991950-0, fax -999, www.genua.de Geschaeftsfuehrer: Dr. Magnus Harlander, Dr. Michaela Harlander, Bernhard Schneck. Amtsgericht Muenchen HRB 98238