| CC: | imapsync <imapsync [...] linux-france.org> |
| Subject: | [Mail-IMAPClient] Memory consumption with message_string() and append() |
| Date: | Mon, 13 Sep 2010 06:44:11 +0200 |
| To: | bug-Mail-IMAPClient [...] rt.cpan.org |
| From: | Gilles LAMIRAL <gilles.lamiral [...] laposte.net> |
Hello,
With message_string() Mail-IMAPClient consumes memory about 7 or 8 times a message size.
With append() Mail-IMAPClient consumes memory about 7 or 8 times a message size.
Example:
For a simple transfer done with this code:
...
my $string = $imap->message_string($msg);
$imap->append('INBOX.Trash', $string);
The memory consumed with this simple code is about 18 times the size of the message, the size of $string here.
To transfer a big message of 50 MB the machine need at least 900 MB of RAM.
I attached a perl script to show memory consumption on a Unix machine.
Just put a big message (10 MB) in folder INBOX.bigmail and run it like this:
./memory_consumption 127.0.0.1 user user_password INBOX.bigmail
Output:
./memory_consumption 127.0.0.1 user user_password INBOX.bigmail
1
main, ./memory_consumption, 31, main::memory_consumption
PID %CPU COMMAND VSZ RSS SZ
18979 0.0 perl 9192 5072 3312
--------------------------------------------------------------------------------
size 1 14370501
main, ./memory_consumption, 36, main::memory_consumption
PID %CPU COMMAND VSZ RSS SZ
18979 37.0 perl 121480 117368 115600
||||||
after message_string-------^^^^^^
--------------------------------------------------------------------------------
main, ./memory_consumption, 38, main::memory_consumption
PID %CPU COMMAND VSZ RSS SZ
18979 66.8 perl 261932 257888 256052
||||||
after append---------------^^^^^^
...
261932000/14370501 = 18
Why Mail-IMAPClient allocates so much memory for just a string copy?
I can understand 2 or even 3 times allocation size but 8 is quite a big factor.
I have no patch yet to reduce this memory consumption.
Thanks in advance if you can explain or do something with this issue.
==== Numbers
perl v5.10.0 or v5.8.8
uname -a
Linux plume 2.6.18-6-k7 #1 SMP Fri Feb 19 23:58:00 UTC 2010 i686 GNU/Linux
Linux petite 2.6.28-19-generic #64-Ubuntu SMP Wed Aug 18 20:55:57 UTC 2010 i686 GNU/Linux
Mail-IMAPClient-3.25
--
Au revoir, 09 51 84 42 42
Gilles Lamiral. France, Baulon (35580) 06 20 79 76 06
#!/usr/bin/perl
use warnings;
use strict;
use English;
use Mail::IMAPClient;
$ARGV[3] or die "usage: $0 host user password folder\n";
my $host = $ARGV[0];
my $user = $ARGV[1];
my $password = $ARGV[2];
my $folder = $ARGV[3];
my $imap = Mail::IMAPClient->new();
$imap->Debug(0);
$imap->Server($host);
$imap->connect() or die;
$imap->User($user);
$imap->Password($password);
$imap->login() or die;
$imap->Uid(1);
$imap->Peek(1);
$imap->Clear(1);
#print map {"$_\n"} $imap->folders();
$imap->select($folder) or die;
my @msgs = $imap->messages or die "Could not messages: $@\n";
print "@msgs\n";
print memory_consumption();
foreach my $msg (@msgs) {
my $size = $imap->size($msg);
print "size $msg $size\n";
my $string = $imap->message_string($msg);
print memory_consumption();
$imap->append('INBOX.Trash', $string);
print memory_consumption();
}
$imap->close();
print memory_consumption();
sub memory_consumption {
my @PID = (@_) ? @_ : ($PROCESS_ID);
my $val;
my ($package, $filename, $line, $subroutine) = caller(0);
$val = "$package, $filename, $line, $subroutine\n";
$val .= qx{ ps o pid,pcpu,comm,vsz,rss,size @PID };
$val .= '-' x 80 . "\n";
return($val);
}