Skip Menu |

This queue is for tickets about the Mail-Box CPAN distribution.

Report information
The Basics
Id: 61460
Status: resolved
Priority: 0/
Queue: Mail-Box

People
Owner: Nobody in particular
Requestors: avar [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Normal
Broken in: 2.095
Fixed in: (no value)



Subject: Mail::Box uses a lot of memory to list a Maildir
This is probably a general problem with Mail::Box but I've only tried this with a Maildir. When I run a simple program to list a Maildir (attached) which just iterates over $folder->messages and calls some methods on the $message object memory use seems to grow as Mail::Box processes more messages. There's probably a memory leak somewhere in the module that's causing this. E.g. on a Maildir with 35k messages it consumed around 800MB of RES memory on Linux, and on a smaller Maildir with ~15k messages it consumed around 200MB. I've attached a small lsmail.pl which is the troublesome program. It takes a path to a Maildir. lsmail.sh is a small wrapper I use to list all the maildirs in ~/Maildir I've been using this on a Maildir synced with offlineimap from GMail.
Subject: lsmail.pl
#!/usr/bin/env perl use 5.012; use Mail::Box::Manager; use Format::Human::Bytes; my $mailbox = shift || die "no mailbox"; my $mgr = Mail::Box::Manager->new; my @open_options = ( access => 'r', extract => 'LAZY' ); my $folder = $mgr->open(folder => $mailbox, @open_options); my $total_size = 0; my $total_messages = 0; $mailbox =~ s[/home/avar][~]; for my $message ($folder->messages) { my $subject = $message->head->get('subject') || ''; my $from = $message->head->get('from') || ''; my $to = $message->head->get('to') || ''; my $list = $message->head->get('list-id') || ''; my $size = $message->size; my $size_human = Format::Human::Bytes::base2($size); printf "%10d\t%s\tsubject:<%s>\tfrom:<%s>\tto:<%s>\tlist:<%s>\tmailbox:<%s>\n", $size, $size_human, $subject, $from, $to, $list, $mailbox; # totals $total_size += $size; $total_messages += 1; } printf "%10d\t%s\tmessages:<%d>\taverage message size:<%s>\tmailbox:<%s>\n", $total_size, Format::Human::Bytes::base2($total_size), $total_messages, Format::Human::Bytes::base2($total_size / ($total_messages||1)), $mailbox;
Subject: lsmail.sh
Download lsmail.sh
application/x-shellscript 260b

Message body not shown because it is not plain text.

Subject: Re: [rt.cpan.org #61460] Mail::Box uses a lot of memory to list a Maildir
Date: Sun, 19 Sep 2010 15:54:27 +0200
To: AEvar Arnfjord Bjarmason via RT <bug-Mail-Box [...] rt.cpan.org>
From: NLnet webmaster <webmaster [...] nlnet.nl>
* AEvar Arnfjord Bjarmason via RT (bug-Mail-Box@rt.cpan.org) [100919 13:22]: Show quoted text
> Sun Sep 19 09:22:28 2010: Request 61460 was acted upon. > Transaction: Ticket created by AVAR > Queue: Mail-Box > Subject: Mail::Box uses a lot of memory to list a Maildir > Broken in: 2.095 > > This is probably a general problem with Mail::Box but I've only tried > this with a Maildir. > > When I run a simple program to list a Maildir (attached) which just > iterates over $folder->messages and calls some methods on the $message > object memory use seems to grow as Mail::Box processes more messages. > There's probably a memory leak somewhere in the module that's causing > this.
MailBox opens folders lazy, which means that initially it only counts the number of messages and reads the Maildir label index. When you need a message, it will load it from file and then keep it in memory till the folder is closed. So, this may seem a memory-leak but it isn't. There are a number of tests which are designed to detect memory- leaks (from experiences in the past) Show quoted text
> E.g. on a Maildir with 35k messages it consumed around 800MB of RES > memory on Linux, and on a smaller Maildir with ~15k messages it consumed > around 200MB.
Doesn't seem an unfair memory usage to me... especially the headers consume a lot of scalars. Really a lot, for each message. There are two simple ways to combat memory usage. You can $msg->destroy to throw its administration away. Or you can simly reopen the folder for every 1000 messages you have processed. -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net
not an error