| Subject: | Slow handling of large emails |
POE::Component::Client::SMTP is struggling on larger email messages. We
were using .18 successfully for quite some time. After upgrading to .22
the problem started.
To reproduce (you need a large email message on disk > 3MB ):
use POE::Component::Client::SMTP;
# spawn a session
POE::Component::Client::SMTP->send(
From => 'djcornell@gmail.com',
To => [
'djcornell@gmail.com',
],
MessageFile => 'largemsg.eml',
Server => 'gmail-smtp-in.l.google.com',
Alias => 'pococlsmtpX',
Debug => 1,
);
POE::Kernel->run();
After a bit of investigation it appears there is a problem with
POE::Filter::Line (or how this module uses it). Lines 140 and 149 of
POE::Filter::Line perform a substitution on the _ENTIRE_ framing buffer
which in our case is the full 3MB message. An strace on the process
during this procedure the process is doiung tons of system calls to
mmap2 and munmap:
-- SNIP ---
sub get_one {
my $self = shift;
# Process as many newlines an we can find.
LINE: while (1) {
# Autodetect is done, or it never started. Parse some buffer!
unless ($self->[AUTODETECT_STATE]) {
DEBUG and warn unpack 'H*', $self->[INPUT_REGEXP];
last LINE
unless $self->[FRAMING_BUFFER] =~ s/^(.*?)$self->[INPUT_REGEXP]//s;
DEBUG and warn "got line: <<", unpack('H*', $1), ">>\n";
return [ $1 ];
}
# Waiting for the first line ending. Look for a generic newline.
if ($self->[AUTODETECT_STATE] & AUTO_STATE_FIRST) {
last LINE
unless $self->[FRAMING_BUFFER] =~ s/^(.*?)(\x0D\x0A?|\x0A\x0D?)//;
my $line = $1;
---SNIP---
I have briefly looked at the source and have not had time to create a
patch but I imagine it will involve loading the file chunk by chunk to
avoid using too much memory on the system and Filter::Line from toss it
around in memory. If you would like me to attempt a patch let me know
and I'll take a stab at it.
Thanks!
Dave
perl -v: This is perl, v5.8.8 built for i686-linux
uname -a: Gentoo Linux SMP Mon Aug 17 22:19:18 UTC 2009 i686 Intel(R)
Xeon(R) CPU L5420 @ 2.50GHz GenuineIntel GNU/Linux