Bug #30350 for MARC-Charset: Table.pag should be that big?

Mon Oct 29 19:46:03 2007 http://jesteves.myopenid.com/ - Ticket created

Subject:

Table.pag should be that big?

Hi. I was trying to install MARC::Charset in one of my servers, but the process failed with a "No space left on device" message. I traced the problem to the fact that lib/lib/MARC/Charset/Table.pag has more that 400 MiB!! So, should it really be that big? And, if so, this maybe isn't a bug, but I guess people would appreciate an informative message while compiling the module. Nor the README or the output of the "make" says that so much space would be needed. The only thing written to the terminal was "please be patient". Greetings, Julián

Mon Oct 29 20:37:45 2007 mrylander [...] gmail.com - Correspondence added

From:

mrylander [...] gmail.com

On Mon Oct 29 19:46:03 2007, http://jesteves.myopenid.com/ wrote: Show quoted text

> Hi. > > I was trying to install MARC::Charset in one of my servers, but the > process failed with a "No > space left on device" message. > > I traced the problem to the fact that lib/lib/MARC/Charset/Table.pag > has more that 400 MiB!! > > So, should it really be that big? And, if so, this maybe isn't a bug, > but I guess people would > appreciate an informative message while compiling the module. Nor the > README or the output > of the "make" says that so much space would be needed. The only thing > written to the terminal > was "please be patient".

Not, it's not supposed to grow quite /that/ large. Can you fill in some details about your system, such as Perl version and OS version? Thanks, --miker

Mon Oct 29 20:37:52 2007 The RT System itself - Status changed from 'new' to 'open'

Mon Oct 29 21:24:43 2007 jesteves [...] janium.com - Correspondence added

Subject:	Re: [rt.cpan.org #30350] Table.pag should be that big?
Date:	Mon, 29 Oct 2007 19:24:08 -0600
To:	bug-MARC-Charset [...] rt.cpan.org
From:	Julián Esteves <jesteves [...] janium.com>

Sure Mike, Here are the details you request, and quite a little more... - Operating system: FreeBSD 6.2-RELEASE - Perl version: "perl -v" says: "This is perl, v5.8.8 built for i386- freebsd-64int" - Output of the compile process: # cd MARC-Charset-0.96 # perl Makefile.PL compiling marc8/utf8 database, please be patient Checking if your kit is complete... Looks good Writing Makefile for MARC::Charset # ls -lh `find . -name Table.pag` -rw-r--r-- 1 root 100 390M Oct 29 20:02 lib/MARC/Charset/Table.pag I have to say that we first installed via CPAN the current version of MARC::Charset (0.98) with similar results. Since I have myself previously installed in my computer version 0.96 and, since I didn't noticed the situation I tried with that older version instead. Then, when posted my previous message, I related it to 0.96 but the situation continues to appear till 0.98. I my machine I have: - Perl "5.8.6 built for darwin-thread-multi-2level" - And size of Table.pag is: $ ls -lh `find /Library/Perl -name Table.pag` -r--r--r-- 1 root admin 390M May 18 18:00 /Library/Perl/ 5.8.6/MARC/Charset/Table.pag And finally, in another machine with Oracle Enterprise Linux, with Perl 5.8.8 we've just installed MARC::Charset 0.98 and now we have a Table.pag file of 391 MiB. Not to add more pressure ;-) but actually those nearly 400 MiB become nearly 1.2 GiB if you take into account the copies of Table.pag in lib/MARC/Charset, blib/lib/MARC/Charset, and finally, when installed, in Perl site's subdirs. Hope all this helps. Julián El 29/10/2007, a las 06:37 PM, Mike Rylander via RT escribió: Show quoted text

> > <URL: http://rt.cpan.org/Ticket/Display.html?id=30350 > > > On Mon Oct 29 19:46:03 2007, http://jesteves.myopenid.com/ wrote:

>> Hi. >> >> I was trying to install MARC::Charset in one of my servers, but the >> process failed with a "No >> space left on device" message. >> >> I traced the problem to the fact that lib/lib/MARC/Charset/Table.pag >> has more that 400 MiB!! >> >> So, should it really be that big? And, if so, this maybe isn't a >> bug, >> but I guess people would >> appreciate an informative message while compiling the module. Nor >> the >> README or the output >> of the "make" says that so much space would be needed. The only >> thing >> written to the terminal >> was "please be patient".

> > Not, it's not supposed to grow quite /that/ large. Can you fill in > some > details about your system, such as Perl version and OS version? > > Thanks, > > --miker

Sun Aug 24 01:28:40 2008 dam [...] modsoftsys.com - Correspondence added

Subject:	[patch] Table.pag should be that big?
From:	dam [...] modsoftsys.com

[please CC me on replies] On Mon Oct 29 20:37:45 2007, MIKERY wrote: Show quoted text

> Not, it's not supposed to grow quite /that/ large. Can you fill in some > details about your system, such as Perl version and OS version?

Hi, As part of the team that maintains the Debian package of MARC-Charset, I am going to apply the attached patch in order to reduce the Table.pag size. The patch replaces SDBM backend with GDBM, trimming the database size to under 6MB (70 times!). Note that the file name of the database changes - it is just "Table" and there is no ".dir" file. All tests pass. The huge Table.pag problem exhibits both with Perl 5.8 and 5.10. All of Debian architectures experience the problem. See http://packages.debian.org/sid/libmarc-charset-perl (scroll down for the list of architectures and respective package sizes). This is present at least since version 0.95 (the first version packaged in Debian). For reference, this issuue is tracked as Debian bug #481505 (http://bugs.debian.org/481505) I hope this helps, dam Debian Perl Group

# replace SDBM database backend with GDBM. SDBM produces an unnecessarily # huge database (400MB), while GDMB gives a more reasonable, 6MB database. # http://bugs.debian.org/481505 # http://rt.cpan.org/Ticket/Display.html?id=30350 # Author: Niko Tyni --- a/lib/MARC/Charset/Table.pm +++ b/lib/MARC/Charset/Table.pm @@ -34,7 +34,7 @@ UCS code point. These keys map to a seri use strict; use warnings; use POSIX; -use SDBM_File; +use GDBM_File; use MARC::Charset::Code; use MARC::Charset::Constants qw(:all); use Storable qw(freeze thaw); @@ -49,7 +49,7 @@ sub new { my $class = shift; my $self = bless {}, ref($class) || $class; - $self->_init(O_RDONLY); + $self->_init(&GDBM_READER); return $self; } @@ -174,7 +174,7 @@ sub brand_new { my $class = shift; my $self = bless {}, ref($class) || $class; - $self->_init(O_CREAT|O_RDWR); + $self->_init(&GDBM_WRCREAT); return $self; } @@ -184,7 +184,7 @@ sub brand_new sub _init { my ($self,$opts) = @_; - tie my %db, 'SDBM_File', db_path(), $opts, 0644; + tie my %db, 'GDBM_File', db_path(), $opts, 0644; $self->{db} = \%db; }

Tue Jun 30 15:33:32 2009 http://fc7.myopenid.com/ - Correspondence added

On Sun Aug 24 01:28:40 2008, dam@modsoftsys.com wrote: Show quoted text

> [please CC me on replies] > > On Mon Oct 29 20:37:45 2007, MIKERY wrote:

> > Not, it's not supposed to grow quite /that/ large. Can you fill in some > > details about your system, such as Perl version and OS version?

> > Hi, > > As part of the team that maintains the Debian package of MARC-Charset, I > am going to apply the attached patch in order to reduce the Table.pag

size. Show quoted text

>

<snip> I can confirm that the problem with the huge SDBM file is still present in version 1.0, and that the patch submitted by the Debian team does nicely resolve it. The tests even run a little faster with it than with the original ;) NB: Running perl 5.10.0 on linux i686

Tue Jun 30 16:19:13 2009 esummers [...] cpan.org - Taken

Tue Jun 30 16:26:14 2009 esummers [...] cpan.org - Correspondence added 15 min

Thanks for the GDBM_File patch folks at Debian! Applied to cvs (ach! gotta upgrade that) at sourceforge, and just uploaded as v1.1. to pause.cpan.org.

Tue Jun 30 16:26:18 2009 esummers [...] cpan.org - Status changed from 'open' to 'resolved'