Bug #61806 for Mail-IMAPClient: Major problem with one function in IMAPClient

Thanks for all your hard work! Hi, I'm using your module to multiplex 1,500+ email accounts and it works great except for one issue that I found yesterday. For the last 6 months, since my project has started, my perl script has been using 400+ MB or memory. Now, that was a lot more than I expected, but I was still ok with it since I had the resources. But then it creeps up, the memory usage continues to increase slowly until after a few days I have to restart it. This issue has plagued me for months while I kept thinking about it in the back of my head and continued development, until one day a few weeks ago I sprinkled free memory available statements all over my code and eventually when I narrowed it down, put a statement after every line until I found the offending line that caused the increase. The memory increase did not occur every time the function was called, but when there was a memory increase, it was always immediately after this statement. my @msgs = $imap->unseen(); Now, I wouldn't think that function would cause any significant memory usage since the doc says all it does is retrieve the message id's of unseen messages. Even if someone had hundreds of unread email, it shouldn't be more than a few kilobytes. But I humored myself and went ahead and reconstructed that line of code with different methods since the numbers don't lie. The * only* change I made was commenting that line out and replacing it with the following: # Get unread messages #my @msgs = $imap->unseen(); # 9/27/2010 - Commented out, suspect this method call to be leaking memory, trying other things $imap->Uid(0); # Making sure to use sequence numbers for this even though it doesn't look like its used my $msg_count = $imap->message_count("Inbox") or warn "Error obtaining unread count: $@\n"; my $unread_count = $imap->unseen_count("Inbox") or warn "Error getting unread count for primary account: $@\n"; my $msg = $imap->message_uid($msg_count) or warn "Could not get uid for $msg_count: $@\n"; $imap->Uid(1); I have before and after graphs of memory usage. Before, about a month ago, had less users, but as you can see memory starts higher and slowly rises, this was typical every day: After, taken this morning (9/28/2010), the first half of the graph is still with the original script, as you can see, there are occasional large losses in memory, and then look at the second half with the new version, flatlines *under* 200 MB! HUGE improvement: There must be some case in the "unseen" function that causes such a big use of memory. After my memory use went down, it looks like my cpu use went down as well, probably less paging going on now. I am using Perl v5.12.1. But I also experienced this with 5.8, I upgraded hoping it would solve my problem because I thought it was a perl issue. Could you look at the unseen function when you get time and see if you find anything suspicious? I suspect other people may, or should, be running into this issue as well. $ uname -a Linux ps14788 2.6.35.4-vs2.3.0.36.32-2+ #14 SMP Wed Sep 15 13:13:54 PDT 2010 x86_64 GNU/Linux $ perl -v This is perl 5, version 12, subversion 1 (v5.12.1) built for x86_64-linux-thread-multi $ perl -V Summary of my perl5 (revision 5 version 12 subversion 1) configuration: Platform: osname=linux, osvers=2.6.33.3-vs2.3.0.36.30.4-swap-nooomloop-nolivelock-oom-jt3, archname=x86_64-linux-thread-multi uname='linux ps14788 2.6.33.3-vs2.3.0.36.30.4-swap-nooomloop-nolivelock-oom-jt3 #13 smp wed may 26 15:49:33 pdt 2010 x86_64 gnulinux ' config_args='-Dprefix=/home/ckdmgr/perl-5.12.1-threads-glibc -Dusethreads -Uusemymalloc' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.1.2 20061115 (prerelease) (Debian 4.1.1-21)', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64 libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=/lib/libc-2.3.6.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.3.6' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API Built under linux Compiled at Aug 10 2010 16:41:28 @INC: /home/ckdmgr/perl-5.12.1-threads-glibc/lib/site_perl/5.12.1/x86_64-linux-thread-multi /home/ckdmgr/perl-5.12.1-threads-glibc/lib/site_perl/5.12.1 /home/ckdmgr/perl-5.12.1-threads-glibc/lib/5.12.1/x86_64-linux-thread-multi /home/ckdmgr/perl-5.12.1-threads-glibc/lib/5.12.1 .