Bug #21707 for Memoize: Memoize causes fatal errors in subsequent threads

Hi Mark Jason. My name is Josh Engelman; I'm a software developer at D.E. Shaw & Co. We've noticed a problem with Memoize.pm when used with multiple threads in perl 5.8.6. I'm not sure if you're still maintaining this module; if there's someone else you think we should contact, please let me know. The problem seems to be that function addresses change in the new thread, but the %memotable and %revmemotable keys still refer to the old function addresses. This means that any attempt to use a memoized function from a second thread raises an exception that kills that thread. Here's a very simple demonstration: $ perl -e ' use threads; use Memoize; Memoize::memoize("foo"); sub foo { $_[0] } my $t1 = threads->new(\&foo, 1); printf "%d\n", $t1->join(); ' thread failed to start: Anonymous function called in forbidden scalar context; faulting at -e line 6 0 This is unfortunate, because it means that any use of a memoized function in *any* perl module makes that module unsafe to use in multi-threaded programs. Since Memoize.pm is part of the core perl distribution, this has far-reaching ramifications. I haven't been able to come up with a good way to make Memoize generically thread-safe -- doing so requires re-engineering of the way memotables work, or rebuilding them when the new thread is created. However, a very small patch will allow memoized functions to continue working in subsequent threads. Basically, the idea is to pass the serialized version of $cref to the _memoizer(), rather than passing the reference itself. The reference itself will have a different serialized form in subsequent threads, but we want to continue using the same %memotable key. Here's the patch diff: diff -c /usr/local/lib/perl5/site_perl/5.8.6/Memoize.pm Memoize.pm *** /usr/local/lib/perl5/site_perl/5.8.6/Memoize.pm Fri Jul 12 13:16:00 2002 --- Memoize.pm Mon Sep 25 13:21:24 2006 *************** *** 72,78 **** # goto version everywhere. my $wrapper = $Config{usethreads} ! ? eval "sub $proto { &_memoizer(\$cref, \@_); }" : eval "sub $proto { unshift \@_, \$cref; goto &_memoizer; }"; my $normalizer = $options{NORMALIZER}; --- 72,78 ---- # goto version everywhere. my $wrapper = $Config{usethreads} ! ? eval "sub $proto { &_memoizer('$cref', \@_); }" : eval "sub $proto { unshift \@_, \$cref; goto &_memoizer; }"; my $normalizer = $options{NORMALIZER}; Note that this solution will not fix unmemoize() in subsequent threads, nor address any complications caused by tied caches. But I believe it fixes the most common case. I've attached the full patched module along with a slightly more extensive test script. Note that I've tested both the problem and the solution with perl 5.8.6 on Solaris 10 x86, Solaris 10 sparc, Windows XP, <<Memoize.pm>> a <<break_memoize.pl>> nd MacOS 10.4. Please let me know if I can provide any further information or assistance. Thx. -josh === Josh Engelman Vice President, D.E. Shaw & Co. Ph. 212.478.0614 Fax 212.845.1614