Bug #18982 for POE: Memory leak

First, I only have some of the components left as bite size test scripts, but I have included those below. I will try to recreate more as I have time, which is none atm. Now, I have narrowed a couple things down and have an explanation as to why you don't see the leak. Here is a break down of my findings so far: Show quoted text
serialization package from INC within POE::Filter::Reference. Show quoted text
POE::Filter::Reference->new() is called. Show quoted text
available via ppm) has the leak when using POE::Component::Server::TCP. Show quoted text
haven't verified, but my guess is that the filter is created once and then re-used. I don't know how deep in this fix goes (ie., Client::TCP, SocketFactory + Wheels w/filter). While this means that POE::Component::Server might be ok (now - and some yet to be determined by me subset of packages), there is the possibility that people will be bitten by this leak using POE::Filter::Reference outside of POE::Component::Server::TCP. It also means that anyone doing something like the following will have serious problems (which includes me of course...): POE::Session->create( inline_states => { _start => sub { &server_start($self,@_); }, server_accepted => sub { &server_accepted($self,@_); }, server_error => sub { &server_error($self,@_); }, client_input => sub { &client_input($self,@_); }, client_error => sub { &client_error($self,@_); }, client_flush => sub { &client_error($self,@_); }, }, ); sub server_start { my $self = shift; $_[HEAP]->{server} = POE::Wheel::SocketFactory->new( BindPort => $self->{port}, SuccessEvent => "server_accepted", FailureEvent => "server_error", ); } sub server_accepted { my $wheel = POE::Wheel::ReadWrite->new( Handle => $client_socket, InputEvent => "client_input", ErrorEvent => "client_error", ); $wheel->set_filter( POE::Filter::ReferenceFix->new("YAML") ); } I have attached two test files, but they are now of limited use since: Show quoted text
server. The only one that should leak now is the client in "filter" test mode. Show quoted text
accidentaly cleaned it up. The real question is why is this delete happening anyway and is that worth anyone using POE::Filter::Reference running the risk of memory leaking? It isn't exactly helping with cleanup, it slows down the filter by forcing a reload of the package, and then challenging the universe to break something because your re-BEGINing a module (which, at a minimum is causing warnings to spew out on packages like YAML). Why is the the delete from INC there? On Sat Apr 29 20:36:09 2006, RCAPUTO wrote: Show quoted text
but Show quoted text
related Show quoted text
linux Show quoted text
http://poe.perl.org/? Show quoted text
to Show quoted text
client Show quoted text
leak Show quoted text
then Show quoted text
of Show quoted text
the Show quoted text
else Show quoted text
based Show quoted text

#!/usr/bin/perl use warnings; use strict; use Storable; use YAML; use POE; use POE::Filter::Reference; use POE::Component::Server::TCP; use Time::HiRes qw(time); use Data::Dumper; our $DEBUG = 0; POE::Component::Server::TCP->new ( Alias => "sum_server", Address => "localhost", Port => 54321, ClientFilter => [ "POE::Filter::Reference","YAML" ], # Handle client requests here. ClientInput => sub { my ( $heap, $ref ) = @_[ HEAP, ARG0 ]; $DEBUG && print "D: ".Dumper($ref)."\n"; my $inc = $ref->[0]->{'Inc'}; my $ctime = $ref->[0]->{'Time'}; print "Client said: $inc , $ctime \n"; $heap->{client}->put({ Inc => $inc , ClientTime => $ctime , ServerTime => time() , }); }, ClientDisconnected => sub { my $kernel = $_[KERNEL]; print "ClientDisconnected\n"; $kernel->yield("shutdown"); }, ); $poe_kernel->run();

#!/usr/bin/perl use strict; use warnings; use YAML; use IO::Socket; use Data::Dumper; use Time::HiRes qw(time); use POE::Filter::Reference; $testcase = 'raw'; for (1 .. 100000) { if ($testcase = 'raw') { my (@resp) = callYAML('127.0.0.1', { 'Inc' => $_, 'Time' => time(), } ); } else { my (@resp) = callFilter('127.0.0.1', { 'Inc' => $_, 'Time' => time(), } ); } print "Response $_ : ".Dumper(\@resp)."\n"; } sub callYAML { #============================================================================= #| #| Use: $response = $o->call(action,@params); #| Info: Call into an zPlay RPC interface. @params is determined by the #| format required by the function you are calling. #| #============================================================================= my $host = shift; my $sock = IO::Socket::INET->new( PeerAddr => $host, PeerPort => 54321, Proto => 'tcp', Timeout => 3, ); if (! defined $sock) { warn("Unable to create socket to yaml server"); return undef; } $sock->autoflush(1); my $y = YAML::Dump([@_]); $sock->print(length($y).chr(0).$y); my $r = $sock->getline(); my $null = chr(0); my (@lp) = split /$null\-\-\-/,$r; $lp[0] = ($lp[0]*1)-length($r)+length($lp[0])+1; # futz with read lengths...guess here my $buf; $sock->read($buf,$lp[0]); $sock->shutdown(2); return ('RawYaml',YAML::Load($buf)); } sub callFilter { my $host = shift; my $sock = IO::Socket::INET->new( PeerAddr => $host, PeerPort => 54321, Proto => 'tcp', Timeout => 3, ); if (! defined $sock) { warn("Unable to create socket to yaml server"); return undef; } $sock->autoflush(1); my $filter = POE::Filter::Reference->new("YAML"); my $y = $filter->put([\@_]); $sock->print($y->[0]); my $r = $sock->getline(); my $null = chr(0); my (@lp) = split /$null\-\-\-/,$r; $lp[0] = ($lp[0]*1)-length($r)+length($lp[0])+1; # futz with read lengths...guess here my $buf; $sock->read($buf,$lp[0]); $sock->shutdown(2); return ('Poe::Filter',$r); }

The problem and the solution are dead simple... However, I haven't found a decent way to detect this memory leak from within the running process itself, especially in an OS neutral way (if there really is one), so while I have an example of the problem (attached) I don't have a proper test case atm. Any suggestions are welcome, although due to the simplicity of the problem (AND SOLUTION), I really think this might be overkill anyway. As it stands, you have to let it run and use top or task manager to watch the memory top the charts of "Greatest Memory Hits of 2006". The code in question is: POE/Filter/Reference.pm 100: $package =~ s(::)(\/)g; 101: delete $INC{$package . ".pm"}; 102: 103: eval { 104: local $^W=0; 105: require "$package.pm"; 106: import $freezer (); 107: }; 109: carp $@ if $@; Since this chunk of code is in the new method, I presume it is being done to force a reload of the module, rather than memory savings. The problem, as I guess at it, is that simply removing the package name from the %INC table (which is just a "record" keeping device...no?) is not cleaning up any or all of the associated Symbol table data for the module (and who knows what else). Every time a new POE::Filter::Reference is created, regardless if we have already used that Reference Filter, it is forced to reload the underlying module without the deep vodoo of the perl guts noticing that it needs to clean up something (if it even can). The solution is simple! We delete line 101, which reads as: 101: delete $INC{$package . ".pm"}; 100: $package =~ s(::)(\/)g; 101: # LINE DELETED 102: 103: eval { 104: local $^W=0; 105: require "$package.pm"; 106: import $freezer (); 107: }; 109: carp $@ if $@; Viola, memory leak is gone! If forcing a module reload is that vital, I suggest following in the footsteps of others: * Symbol.pm using delete_package * ModPerl::Util::unload_module (maybe they did something magic) * http://cpan.uwinnipeg.ca/search?query=reload&mode=dist However, unless having a running (and possibly even running in production) script being able to load updated modules on the fly is an important requirement, I would lose it.

#!/usr/bin/perl use strict; use warnings; use Time::HiRes qw(sleep time); use POE::Filter::Reference; for (1 .. 10000) { my $filter = POE::Filter::Reference->new("Storable"); my $f = $filter->put([ ['Freeze','Me'] ]); my $t = $filter->get($f); sleep(0.01); next; }