Bug #93200 for Git-FastExport: Memory Leak

Thu Feb 20 11:54:31 2014 hector.izquierdo [...] gmail.com - Ticket created

Subject:	Memory Leak
Date:	Thu, 20 Feb 2014 16:54:23 +0000
To:	bug-Git-FastExport [...] rt.cpan.org
From:	Héctor Izquierdo Seliva <hector.izquierdo [...] gmail.com>

Hi, I'm trying to use git-stitch-repo with two big repositories (9.8 and 3.4 gigs respectively). What I've found is that the perl process memory usage keeps rising and rising, until all the memory in the system is exhausted. System: Unbutu 13.10 x86_64 Perl: v5.14.2 Git: 1.8.3.2 Let me know if I can provide any information that may help you track the error. Hector

Sat Feb 22 18:11:18 2014 book [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #93200] Memory Leak
Date:	Fri, 21 Feb 2014 09:13:19 +0100
To:	Héctor Izquierdo Seliva via RT <bug-Git-FastExport [...] rt.cpan.org>
From:	"Philippe Bruhat (BooK)" <book [...] cpan.org>

On Thu, Feb 20, 2014 at 11:54:32AM -0500, Héctor Izquierdo Seliva via RT wrote: Show quoted text

> > I'm trying to use git-stitch-repo with two big repositories (9.8 and 3.4 > gigs respectively). What I've found is that the perl process memory usage > keeps rising and rising, until all the memory in the system is exhausted.

I'm really surprised about it, because it should be trying to minimize memory consumption. It does however keep the whole commit lineage in memory each repository... Show quoted text

> Let me know if I can provide any information that may help you track the > error.

How big are the repositories? Are they public? Is there a way for me to reproduce the issue ? -- Philippe Bruhat (BooK) No matter who you may be, there is always someone who is a little worse because he thinks he is a little better. (Moral from Groo The Wanderer #3 (Epic))

Sat Feb 22 18:11:18 2014 The RT System itself - Status changed from 'new' to 'open'

Sat Feb 22 18:21:07 2014 hector.izquierdo [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #93200] Memory Leak
Date:	Sat, 22 Feb 2014 23:20:54 +0000
To:	bug-Git-FastExport [...] rt.cpan.org
From:	Héctor Izquierdo Seliva <hector.izquierdo [...] gmail.com>

I'm afraid the repository is not public. If it's of any help between the two repositories there are around 50k commits. On Sat, Feb 22, 2014 at 11:11 PM, Philippe Bruhat (BooK) via RT < bug-Git-FastExport@rt.cpan.org> wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=93200 > > > On Thu, Feb 20, 2014 at 11:54:32AM -0500, Héctor Izquierdo Seliva via RT > wrote:

> > > > I'm trying to use git-stitch-repo with two big repositories (9.8 and 3.4 > > gigs respectively). What I've found is that the perl process memory usage > > keeps rising and rising, until all the memory in the system is exhausted.

> > I'm really surprised about it, because it should be trying to minimize > memory consumption. It does however keep the whole commit lineage in > memory each repository... >

> > Let me know if I can provide any information that may help you track the > > error.

> > How big are the repositories? Are they public? Is there a way for me > to reproduce the issue ? > > -- > Philippe Bruhat (BooK) > > No matter who you may be, there is always someone who is a little worse > because he thinks he is a little better. > (Moral from Groo The Wanderer #3 > (Epic)) > >

Mon Feb 24 05:56:52 2014 book [...] cpan.org - Correspondence added

On Sat Feb 22 18:21:07 2014, hector.izquierdo@gmail.com wrote: Show quoted text

> I'm afraid the repository is not public. If it's of any help between the > two repositories there are around 50k commits.

OK. I've found one internal data structure that would be using a lot of memory unnecessarily. I've rearranged it to use a little memory as possible instead. Basically, I was recording the whole list of commits in the lineage of any commit, which means that the most recent commits in your repository would carry arround hashes with 50k keys. Even for a fully linear history, that would mean having 125 million hash values floating around... No wonder it was eating up memory! Check out v0.102 on CPAN, and let me know how that works for your repositories ! -- BooK

Mon Feb 24 05:58:53 2014 hector.izquierdo [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #93200] Memory Leak
Date:	Mon, 24 Feb 2014 10:58:44 +0000
To:	bug-Git-FastExport [...] rt.cpan.org
From:	Héctor Izquierdo Seliva <hector.izquierdo [...] gmail.com>

Thanks! I'll try it as soon as I can and report back. On 24 Feb 2014 10:56, "Philippe Bruhat (BooK) via RT" < bug-Git-FastExport@rt.cpan.org> wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=93200 > > > On Sat Feb 22 18:21:07 2014, hector.izquierdo@gmail.com wrote:

> > I'm afraid the repository is not public. If it's of any help between the > > two repositories there are around 50k commits.

> > OK. I've found one internal data structure that would be using a lot of > memory unnecessarily. I've rearranged it to use a little memory as > possible instead. > > Basically, I was recording the whole list of commits in the lineage of any > commit, > which means that the most recent commits in your repository would carry > arround hashes > with 50k keys. Even for a fully linear history, that would mean having 125 > million > hash values floating around... No wonder it was eating up memory! > > Check out v0.102 on CPAN, and let me know how that works for your > repositories ! > > -- BooK >

Wed Feb 26 10:10:39 2014 hector.izquierdo [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #93200] Memory Leak
Date:	Wed, 26 Feb 2014 15:10:23 +0000
To:	bug-Git-FastExport [...] rt.cpan.org
From:	Héctor Izquierdo Seliva <hector.izquierdo [...] gmail.com>

Memory consumption is now fine. Thanks for fixing so fast! On Mon, Feb 24, 2014 at 10:58 AM, Héctor Izquierdo Seliva < hector.izquierdo@gmail.com> wrote: Show quoted text

> Thanks! I'll try it as soon as I can and report back. > On 24 Feb 2014 10:56, "Philippe Bruhat (BooK) via RT" < > bug-Git-FastExport@rt.cpan.org> wrote: >

>> <URL: https://rt.cpan.org/Ticket/Display.html?id=93200 > >> >> On Sat Feb 22 18:21:07 2014, hector.izquierdo@gmail.com wrote:

>> > I'm afraid the repository is not public. If it's of any help between the >> > two repositories there are around 50k commits.

>> >> OK. I've found one internal data structure that would be using a lot of >> memory unnecessarily. I've rearranged it to use a little memory as >> possible instead. >> >> Basically, I was recording the whole list of commits in the lineage of >> any commit, >> which means that the most recent commits in your repository would carry >> arround hashes >> with 50k keys. Even for a fully linear history, that would mean having >> 125 million >> hash values floating around... No wonder it was eating up memory! >> >> Check out v0.102 on CPAN, and let me know how that works for your >> repositories ! >> >> -- BooK >>

>

Wed Feb 26 13:27:51 2014 book [...] cpan.org - Status changed from 'open' to 'resolved'

Wed Feb 26 13:27:52 2014 book [...] cpan.org - Fixed in 0.102 added

Wed Feb 26 13:30:02 2014 book [...] cpan.org - Correspondence added

On Wed Feb 26 10:10:39 2014, hector.izquierdo@gmail.com wrote: Show quoted text

> Memory consumption is now fine. Thanks for fixing so fast! >

Glad that it worked, and thanks for the initial report. Since you're one of the few actual users of the tool, would you mind leaving a rating for the module at http://cpanratings.perl.org/dist/Git-FastExport ? Thanks, -- BooK

Fri Feb 28 08:12:06 2014 hector.izquierdo [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #93200] Memory Leak
Date:	Fri, 28 Feb 2014 13:11:56 +0000
To:	bug-Git-FastExport [...] rt.cpan.org
From:	Héctor Izquierdo Seliva <hector.izquierdo [...] gmail.com>

Not at all! Hector On Wed, Feb 26, 2014 at 6:30 PM, Philippe Bruhat (BooK) via RT < bug-Git-FastExport@rt.cpan.org> wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=93200 > > > On Wed Feb 26 10:10:39 2014, hector.izquierdo@gmail.com wrote:

> > Memory consumption is now fine. Thanks for fixing so fast! > >

> > Glad that it worked, and thanks for the initial report. > > Since you're one of the few actual users of the tool, > would you mind leaving a rating for the module at > http://cpanratings.perl.org/dist/Git-FastExport ? > > Thanks, > > -- BooK >