| Subject: | Memoize::Storable needs locking |
If concurrent processes use the same cache file, then it's possible that every now and then reading the cache file fails due to simultaneous writing of it. I think it's also possible that two simultaneously writing processes leave a corrupt storable file.
Storable provides lock variants for retrieve/store/nstore (named lock_retrieve/lock_store/lock_nstore). It would be nice if Memoize::Storable had an option to use these locking variants (say, by specifying locking => 1 or so).
Below is a demonstration script which sometimes fails with the error
Magic number checking on storable file failed at /usr/lib/x86_64-linux-gnu/perl/5.30/Storable.pm line 421, at /usr/share/perl/5.30/Memoize/Storable.pm line 21.
if run simultaneously (try it using "./memoize-storable.pl & ./memoize-storable.pl" a couple of times). The locking in the demonstration script is purely to increase the likelihood of reaching this situation (after unlocking the first process starts to write the cache, at the same time the other process acquires the lock and starts to read it).
#!/usr/bin/perl
use strict;
use warnings;
use Memoize qw(memoize unmemoize);
use Memoize::Storable;
use Fcntl qw(LOCK_EX);
sub function { rand() }
open my $fh, $0 or die $!;
flock $fh, LOCK_EX or die $!;
{
my $filename = '/tmp/memoize.st';
tie my %cache => 'Memoize::Storable', $filename;
memoize 'function', SCALAR_CACHE => [HASH => \%cache];
warn scalar function();
unmemoize 'function';
close $fh;
}
__END__