Bug #4943 for CPAN: CPAN module too memory intensive

Wed Jan 14 22:33:53 2004 Guest - Ticket created

Subject:

CPAN module too memory intensive

I use CPAN.pm on a current Debian installation on a slow older machine without a whole lot of memory, and the step in its operation following the outputting of Going to write /root/.cpan/Metadata takes a real long time. the "top" utility tells me that the process is using over 38M of memory. Would it be possible to alter CPAN so it takes less memory by throwing out intermediate variables earlier, or storing more in the FS rather than in memory? Are there installation / configuration options that I should know about, to optimize my site for little memory, much disk space? (i think a configuration option would make more sense than a CPAN-LOWMEM.pm fork.) Maybe indexing could be moved server-side, instead of downloading and parsing the whole index whenever I want to use the CPAN module to "streamline" installing a single module (which is usually how I use it.) Maybe "CPANcache" daemons could take care of the indexing, and CPAN would be configured to use a CPANcache daemon if available. PM groups could run CPANcache daemons. I know mine would at least. ... Storable->nstore($cache) might be taking a long time, or it might be whatever comes after that. I wonder if the cache was kept on disk in smaller grains, if that would help. (perhaps using DirDB or a variant that could effectively nstore the various CPAN objects, instead of punting the whole issue to Storable.) Another place where CPAN shell appears to hang is on exiting, after "lockfile removed" we don't appear to exit. It's still more convenient than running ftp, gunzip, Make, yadda yadda. thanks

Thu Sep 30 14:37:02 2004 Guest - Correspondence added

From:

t35t0r [...] gmail.com

Hi, I have the same problem but on a VPS (virtual private server, quad xeon processor) over at globalservers.com. CPAN stops during untar of "install Bundle::CPAN" or install of any other module/package. According to the techsupport guy, it was trying to grab in excess of 100MB allocated to the VPS. He gave it 150MB temporarily and the problem went away. Show quoted text

cpan> install Bundle::CPAN

...downloads some files . Using Tar:/bin/tar xvf /root/.cpan/sources/authors/id/G/GB/GBARR/Bundle-libnet-1.00.tar: Couldn't untar /root/.cpan/sources/authors/id/G/GB/GBARR/Bundle-libnet-1.00.tar . . then later on I get this error: [root@someBox ~]# limit cputime unlimited filesize unlimited datasize unlimited stacksize unlimited coredumpsize unlimited memoryuse unlimited vmemoryuse unlimited descriptors 1024 memorylocked unlimited maxproc unlimited [root@someBox ~]# perl -MCPAN -e shell Out of memory! Callback called exit. BEGIN failed--compilation aborted. This is strange because I use untar on the same files with no problems from the tcsh/bash prompt. So my question in general is the same as the original given in the first post, is there something that can be done to reduce the memory usage? CPAN is indeed much more convenient than wget, make, etc.. especially when there are several dependencies which need to be installed before the target package can be installed. Thanks [guest - Wed Jan 14 22:33:53 2004]: Show quoted text

> I use CPAN.pm on a current Debian installation on a slow older machine > without a whole lot of memory, > and the step in its operation following the outputting of > > Going to write /root/.cpan/Metadata > > takes a real long time. the "top" utility tells me that the process > is using over 38M of memory. > > Would it be possible to alter CPAN so it takes less memory by throwing > out intermediate variables earlier, or storing more in the FS rather > than > in memory? > > Are there installation / configuration options that I should know > about, > to optimize my site for little memory, much disk space? > > (i think a configuration option would make more sense than a CPAN- > LOWMEM.pm > fork.) > > Maybe indexing could be moved server-side, instead of downloading > and parsing the whole index whenever I want to use the CPAN module to > "streamline" installing a single module (which is usually how I use > it.) > > Maybe "CPANcache" daemons could take care of the indexing, and CPAN > would be configured to use a CPANcache daemon if available. PM groups > could run CPANcache daemons. I know mine would at least. > > > ... > > > Storable->nstore($cache) might be taking a long time, or it might be > whatever comes after that. > > I wonder if the cache was kept on disk in smaller grains, if that > would help. (perhaps using DirDB or a variant that could effectively > nstore the various CPAN objects, instead of punting the whole issue > to Storable.) > > Another place where CPAN shell appears to hang is on exiting, after > "lockfile removed" we don't appear to exit. > > > It's still more convenient than running ftp, gunzip, Make, yadda > yadda. > > thanks

Thu Jan 26 09:35:14 2006 ANDK [...] cpan.org - Taken

Thu Jan 26 09:38:51 2006 ANDK [...] cpan.org - Status changed from 'new' to 'open'

Fri Jan 05 03:39:37 2007 ANDK [...] cpan.org - Correspondence added

A few weeks ago CPAN::SQLite came out. Developer versions are in RKOBES directory. CPAN.pm 1.88_67 has quite reasonable support for it. With that the memory savings are substantial. On my system from 80 to 20 MB. Memeory usage increases while running though. Please give it a shot. Closing this bug now

Fri Jan 05 03:39:39 2007 ANDK [...] cpan.org - Status changed from 'open' to 'resolved'

Fri Jan 05 12:21:01 2007 whatever [...] davidnicol.com - Correspondence added

Subject:	Re: [rt.cpan.org #4943] CPAN module too memory intensive
Date:	Fri, 5 Jan 2007 11:19:55 -0600
To:	bug-CPAN [...] rt.cpan.org
From:	"David Nicol" <davidnico [...] cpan.org>

that's not exactly how I would have done it, but I am pleased to see this progress. Excellent. I will be congfiguring CPAN this way on the next lowmem box I configure CPAN on. -- pre-Α, Α, Β, rc, release.

Fri Jan 05 12:21:04 2007 The RT System itself - Status changed from 'resolved' to 'open'

Fri Mar 23 03:28:34 2007 ANDK [...] cpan.org - Correspondence added

Although this topic can never really be closed I close the ticket because there's no plan at the moment to further improve on it.

Fri Mar 23 03:28:51 2007 ANDK [...] cpan.org - Status changed from 'open' to 'resolved'