Bug #97508 for Tie-Cache: Tie::Cache together with threads crashes perl

On 2014-07-25 14:15:21, Ralf.Neubauer@wido.bv.aok.de wrote: Show quoted text
The issue is that Tie::Cache is not threads safe. See 'perldoc threads' for more info.

I agree there is a bug here. Sadly I suspect you are right that it is mostly likely in perl's memory management with threads, which I suspect since the segfault gets clear as the range in this line increases: $c{$_} = 1 for 1..30000 30000 seems to be the number that drives this. At 10000 and 20000 I was able to run this fine. Internally the memory use and structures of the Tie::Cache grow as this 30000 range increases and more of the Tie::Cache gets used. Interestingly enough if the size of the Tie::Cache is constrained to smaller size, this code runs: perl -e "use threads; use Tie::Cache; tie %c, 'Tie::Cache', 10000; for(1..30000) { \$c{\$_} = 1 }; (async { print 'Hallo' })->join;" So the Tie::Cache object in this case cannot grow past 10000 elements, so 30000 writes to it does not increase it to 30000 elements. This points further to some thread size memory management issue. I suspect this has to do with a thread stack size running out of space. However I could not prove this, as when I tried to increase thread stack size the problem was not fixed. If its the case, then Tie::Cache might be using upwards of 1K per entry in threaded mode so it seems this is not a good solution under perl threads. On Fri Jul 25 14:15:21 2014, Ralf.Neubauer@wido.bv.aok.de wrote: Show quoted text

On Fri Jul 25 16:56:49 2014, JDHEDDEN wrote: Show quoted text
I believe this is a thread + memory problem... This program runs fine for me outside of a thread context with a larger cache: perl -e "use Tie::Cache; tie %c, 'Tie::Cache', 200000; for(1..1000000) { \$c{\$_} = 1 }; " But there is no thread stack size issue in this case. Do note that the memory overhead here is significant and the program used around 100M memory for 200K has size.

perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..30000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use threads; (async { print qq{Hello\n} })->join' Segmentation fault perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..20000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use threads; (async { print qq{Hello\n} })->join' Hello perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..30000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use Storable qw(dclone); dclone \%c; print qq{Hello\n}' Segmentation fault perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..20000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use Storable qw(dclone); dclone \%c; print qq{Hello\n}' Hello This is with the rather ancient perl 5.14.2-21+deb7u1 i386 on debian. With perl 5.18.2-2ubuntu1 amd64 on ubuntu I get: perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..20000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use Storable qw(dclone); dclone \%c; print qq{Hello\n}' Hello perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..30000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use Storable qw(dclone); dclone \%c; print qq{Hello\n}' Segmentation fault (core dumped) perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..30000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use threads; (async { print qq{Hello\n} })->join' Hello perl -e 'my @a = map [ $_, 0, 1, undef, undef ], 0..40000; for (0..$#a-1) { $a[$_+1][3] = $a[$_]; $a[$_][4] = $a[$_+1] }; my %h = map +($_->[0] => $_), @a; %c = (node => \%h, head => $a[0], tail => $a[$#a]); use threads; (async { print qq{Hello\n} })->join' Segmentation fault (core dumped) Note I had to raise the number of elements a bit for the last test.

$ perl -e 'use threads; $t = $t->[1] = [$t] for 1..40000; (async { print qq{Hello\n} })->join' Segmentation fault (core dumped) $ perl -e 'use Storable qw(dclone); $t = $t->[1] = [$t] for 1..40000; dclone $t' Segmentation fault (core dumped) $ perl -e 'use Clone qw(clone); $t = $t->[1] = [$t] for 1..40000; clone $t' Segmentation fault (core dumped) $ ulimit -s 8192 $ ulimit -s 16384 $ perl -e 'use Clone qw(clone); $t = $t->[1] = [$t] for 1..40000; clone $t' $ perl -e 'use Clone qw(clone); $t = $t->[1] = [$t] for 1..80000; clone $t' Segmentation fault (core dumped) If for some reason no non-recursive algorithm is possible, an error message 'clone: out of memory' or 'recursion limit reached' would be much nicer than a segfault, which gives no hint about which part of the program crashed (took a while to find out the async {} in a big program crashed because of some object I allocated and filled somewhere completely else). Ok, this is completely off topic for Tie::Cache... Just wanted to write this down for whoever is interested in this bug.

This old bug report seems to be resolved with more recent versions of perl! No longer and issue, and not a bug with Tie::Cache in any case as was documented in bug report.