Skip Menu |

This queue is for tickets about the Net-Async-HTTP CPAN distribution.

Report information
The Basics
Id: 102547
Status: resolved
Priority: 0/
Queue: Net-Async-HTTP

People
Owner: Nobody in particular
Requestors: WolfSage [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: 0.37
Fixed in: 0.38



Subject: do_request(on_error => sub {...}) leaks when there are more than max_connections_per_host attempts
Attached is a script that leaks indefinitely. Every second it makes 6 requests on a Net::Async::HTTP object with max_connections_per_host set to 5. Every few seconds the memory goes up. (To see this happen faster, create 20 connections a second instead of 6). If I load this up under Devel::MAT I see that there are many futures created and many Net::Async::HTTP::Ready/Net::Async::HTTP::Connection objects that are just lying around for some reason. If I do: my $old; my $count; BEGIN { $old = \&Future::new; } { package Future; sub new { $count++; return $old->(@_); } DESTROY { $count--; warn "Count: $count\n"; } } I see $count climb a few times every second when we create 6 requests at a time. If we create <= 5 requests at a time, $count stays at 0 at the end of every cycle. -- Matthew Horsfall (alh)
Subject: leak2.pl
#!/usr/bin/perl use strict; use warnings; use HTTP::Request; use IO::Async::Loop; use IO::Async::Timer::Periodic; use Net::Async::HTTP; use Future; my $loop = IO::Async::Loop->new; my $timer = IO::Async::Timer::Periodic->new( interval => 1, first_interval => 0, on_tick => \&work, ); $timer->start; $loop->add($timer); $timer = IO::Async::Timer::Periodic->new( interval => 60, on_tick => sub { exit }, ); $timer->start; $loop->add($timer); my $http = Net::Async::HTTP->new( max_connections_per_host => 5, pipeline => 0, ); $loop->add($http); $loop->run; sub work { for (1..6) { # Stops leaking if we change this to 5 $http->do_request( # I see 'connection refused' errors for localhost:10234 request => HTTP::Request->new('GET', 'http://127.0.0.1:10234'), on_response => sub { warn "$$: Got response! @_\n"; }, on_error => sub { warn "$$: Got error: @_\n"; }, ); } }
On Fri Mar 06 11:55:44 2015, WOLFSAGE wrote: Show quoted text
> I see $count climb a few times every second when we create 6 requests > at a time. If we create <= 5 requests at a time, $count stays at 0 at > the end of every cycle. > > -- Matthew Horsfall (alh)
So, it turns out I was half right. There's actually a leak in both cases, one is just slower. This is because $conn->remove_from_parent wasn't getting called in the case of connection failures. Attached is a patch that contains a test and a fix demonstrating the issue. Cheers, -- Matthew Horsfall (alh)
Subject: leak-patch.diff
diff -urN Net-Async-HTTP-0.37/lib/Net/Async/HTTP.pm Net-Async-HTTP-0.37-patched/lib/Net/Async/HTTP.pm --- Net-Async-HTTP-0.37/lib/Net/Async/HTTP.pm 2014-12-13 10:08:07.000000000 -0500 +++ Net-Async-HTTP-0.37-patched/lib/Net/Async/HTTP.pm 2015-03-09 10:28:05.619033830 -0400 @@ -419,6 +419,7 @@ $f->fail( @_ ) unless $f->is_cancelled; + $conn->remove_from_parent; @$conns = grep { $_ != $conn } @$conns; @$ready_queue = grep { $_ != $ready } @$ready_queue; diff -urN Net-Async-HTTP-0.37/t/90rt102547.t Net-Async-HTTP-0.37-patched/t/90rt102547.t --- Net-Async-HTTP-0.37/t/90rt102547.t 1969-12-31 19:00:00.000000000 -0500 +++ Net-Async-HTTP-0.37-patched/t/90rt102547.t 2015-03-09 10:27:50.727022971 -0400 @@ -0,0 +1,59 @@ +#!/usr/bin/perl + +use strict; +use warnings; + +use Test::More; +use IO::Async::Test; +use IO::Async::Loop; + +use Net::Async::HTTP; + +# When connections failed, they weren't being removed from memory +# so we'd slowly leak + +my $loop = IO::Async::Loop->new; +testing_loop( $loop ); + +my $http = Net::Async::HTTP->new( + max_connections_per_host => 2, +); + +$loop->add( $http ); + +my @conn_f; +my @remove_f; + +no warnings 'redefine'; +local *IO::Async::Loop::connect = sub { + shift; + my %args = @_; + $args{host} eq "localhost" or die "Cannot fake connect - expected host 'localhost'"; + $args{service} eq "5000" or die "Cannot fake connect - expected service '5000'"; + + push @conn_f, my $f = $loop->new_future; + return $f; +}; + +my $old = \&IO::Async::Notifier::remove_from_parent; + +# Make sure these actually get removed! +local *IO::Async::Notifier::remove_from_parent = sub { + my $self = shift; + push @remove_f, $self; + return $old->($self, @_); +}; + +my @f = map { $http->do_request(uri=>'http://localhost:5000/') } 0 .. 2; + +is( scalar @conn_f, 2, 'Two pending connect() attempts after two concurrent ->do_request' ); + +# Fail them all +( shift @conn_f )->fail( "Connection refused", connect => ) for 0 .. 2; + +ok( !@conn_f, 'No more pending connect() attempts' ); + +is( scalar @remove_f, 3, 'Three connect() attempts removed after connection failure' ); + +done_testing; +
Thanks; patch applied. -- Paul Evans