Skip Menu |

This queue is for tickets about the Math-Prime-Util CPAN distribution.

Maintainer(s)' notes

The latest version is in https://github.com/danaj/Math-Prime-Util

    git clone --origin upstream git://github.com/danaj/Math-Prime-Util.git Math-Prime-Util

Comments, issues, and patches are welcome at either RT or github, or send me email.

Report information
The Basics
Id: 127605
Status: resolved
Priority: 0/
Queue: Math-Prime-Util
People
Owner: DANAJ [...] cpan.org
Requestors: justincase [...] yopmail.com
Cc:
AdminCc:
Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)

History Show all quoted text
  Thu Nov 08 15:38:43 2018 justincase [...] yopmail.com - Ticket created
Subject: Memory leak in forsetproduct
This is a similar bug to that present in Algorithm::Permute and Algorithm::FastPermute: https://rt.cpan.org/Ticket/Display.html?id=126027#txn-1799090 forsetproduct calls it's block repeatedly using MULTICALL, but if the block creates any temporary variables, they are not cleaned up until after forsetproduct returns. This can result in OOM (out of memory). It can be resolved by placing the call to MULTICALL in a pseudoblock, i.e. enclosing it in ENTER/LEAVE.
  Thu Nov 08 16:10:08 2018 DANAJ [...] cpan.org - Taken
  Thu Nov 08 16:38:00 2018 DANAJ [...] cpan.org - Correspondence added
On Thu Nov 08 15:38:43 2018, justincase wrote: Show quoted text
> This is a similar bug to that present in Algorithm::Permute and > Algorithm::FastPermute: > https://rt.cpan.org/Ticket/Display.html?id=126027#txn-1799090 > > forsetproduct calls it's block repeatedly using MULTICALL, but if the > block creates any temporary variables, they are not cleaned up until > after forsetproduct returns. This can result in OOM (out of memory). > It can be resolved by placing the call to MULTICALL in a pseudoblock, > i.e. enclosing it in ENTER/LEAVE.
Set::Product::XS would have the same bug. It is also present in every XS iterator in MPU. If I understand correctly, this isn't so much a leak as it is a mass free when the iterator exits. Which might be a very long time. I believe the builtin map {} function has the same issue, or at least it did a year ago. It requires a little cleverness to make sure the effect isn't dwarfed by the input list. A test for this ticket: perl -Mntheory=:all -E 'my @array=([0..50],[1..100],[1..100],[1..100]); for (1..100000) { forsetproduct { my $str=join "",@_; } @array; }' shows this grow to ~200MB over ~30 seconds then hold steady. Using the ENTER/LEAVE wrapper sees it right away at ~7MB and hold steady. Preliminary fix on github.
  Thu Nov 08 16:38:00 2018 The RT System itself - Status changed from 'new' to 'open'
  Thu Nov 08 16:57:58 2018 justincase [...] yopmail.com - Correspondence added
On Thu Nov 08 16:38:00 2018, DANAJ wrote: Show quoted text
> On Thu Nov 08 15:38:43 2018, justincase wrote:
> > This is a similar bug to that present in Algorithm::Permute and > > Algorithm::FastPermute: > > https://rt.cpan.org/Ticket/Display.html?id=126027#txn-1799090 > > > > forsetproduct calls it's block repeatedly using MULTICALL, but if the > > block creates any temporary variables, they are not cleaned up until > > after forsetproduct returns. This can result in OOM (out of memory). > > It can be resolved by placing the call to MULTICALL in a pseudoblock, > > i.e. enclosing it in ENTER/LEAVE.
> > Set::Product::XS would have the same bug. It is also present in every > XS iterator in MPU.
I also contacted the author of Set::Product::XS and a fix for that is forthcoming. Apparently it was fixed in a previous version and got reverted fixing a different bug. Show quoted text
> If I understand correctly, this isn't so much a leak as it is a mass > free when the iterator exits. Which might be a very long time.
That's correct. The memory can grow very quickly and never reach the point when it would be freed.
  Thu Nov 15 16:56:32 2018 DANAJ [...] cpan.org - Correspondence added
Fixed in v0.73. All MULTICALL operations have been wrapped as { ENTER; MULTICALL; LEAVE; } Thank you again for the timely and excellent report! It is not unlikely this still impacts the non-multicall path. There is a TODO entry for that added.
  Thu Nov 15 16:56:33 2018 DANAJ [...] cpan.org - Status changed from 'open' to 'resolved'