| Subject: | Remove for loop with temp array because of high memory consumption in Flock::_get_lock |
I'm using IPC::ConcurrencyLimit::WithStandby in a scenario where I want to limit access to some resources to exactly one process, while I want ALL other processes to wait until the one currently accessing the resource is finished. WithStandby is already providing all the logic I need, if I only set a high enough value for standby_max_procs. In my case the practical limit is some dozen to some hundred concurrent processes accessing the resource only for few seconds, but just to be sure I defined a massive number for the limit: 99_999_999
During production I recently realized that my processes are sometimes killed by the OOM-killer of Linux, investigated that further and found the problem in IPC::ConcurrencyLimit::Lock::Flock::_get_lock:
Show quoted text
> for my $worker_id (reverse 1 .. $self->{max_procs})
This loop is creating a temporary array and depending on standby_max_procs may consume a massive amount of memory, exhausting the memory resources of my production VM and so OOM-killer kills the process. This can easily be testet using the following code:
Show quoted text> use strict;
> use warnings;
Show quoted text> my @array = (1 .. 99_999_999);
Save that to a file, run "perl -C yourFile.pl" on the shell and have a look at the memory consumption of that process, in my Windows x64 it was at most 7 GB. I hope you agree that this is far to much memory for the use case of the number.
So I suggest to change that for loop to a simpler one which just decrements a temporary counter. Something like the following:
Show quoted text> my $counter = $self->{max_procs};
> while ($counter-- >= 1)
> [...]
While reducing my 99_999_999 to 9_999_999, one less 9, works, some versions of Perl still consume a significant amount of memory in such cases as well[1]. Yes, your loop looks far nicer, but I think the downsides are important enough to change it.
Thanks!
[1]: http://stackoverflow.com/questions/6440723/perl-array-of-integers-using-way-too-much-memory