Skip Menu |

This queue is for tickets about the POE-Component-Resolver CPAN distribution.

Report information
The Basics
Id: 76987
Status: resolved
Priority: 0/
Queue: POE-Component-Resolver

People
Owner: Nobody in particular
Requestors: kozunov [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: (no value)
Fixed in: 0.916



Subject: Resolver doesn't delete request when timeout occure in HTTP::Client
When resolver (by different reason) don't resolve quickly, and timeout fire in Component::Client::HTTP I receive response with 408, but program doesn't exit. Because resolver doesn't delete request.
Subject: for_cpan2.pl
#!/usr/bin/perl -w use utf8; use strict; use warnings; use Data::Dumper; use POE 1.350; use POE::Component::Client::HTTP 0.944; use POE::Component::Client::Keepalive 0.270; use HTTP::Request::Common qw(GET HEAD); use HTTP::Cookies; $SIG{__WARN__} = sub { my $str = shift; warn scalar(localtime)." $str"; }; my $pool = POE::Component::Client::Keepalive->new( keep_alive => 3, max_open => 2, max_per_host => 2, timeout => 7, resolver => POE::Component::Resolver->new( max_resolvers => 1, idle_timeout => 11, sidecar_program => [ '/usr/bin/perl', (map { "-I$_" } @INC), '-MPOE::Component::Resolver::Sidecar', '-e', 'sleep(25); POE::Component::Resolver::Sidecar->main()' ] ), ); POE::Component::Client::HTTP->spawn( Alias => 'ua', FollowRedirects => 3, Timeout => 1, MaxSize => 1024, Agent => '', ConnectionManager => $pool, Streaming => 0, CookieJar => HTTP::Cookies->new(), ); POE::Session->create( inline_states => { _start => sub { my $header = [TE => 'chunked,identity', Connection => "close"]; my $domain = 'localhost:36333'; my $url = "http://$domain/"; my $request = HTTP::Request->new( 'GET', $url, $header); $poe_kernel->post( "ua" => "request", "response", $request, {domain=>$domain}); }, _stop => sub {}, response => \&response, }, ); POE::Kernel->run(); warn 1; exit; sub response { my ($request_packet, $response_packet) = @_[ARG0, ARG1]; my $http_request = $request_packet->[0]; my $tag = $request_packet->[1]; my $http_response = $response_packet->[0]; warn $http_response->status_line; #warn Dumper $tag; #warn Dumper $http_response; }
I think this is probably the same as ticket 76550. The "localhost" domain will be resolved through /etc/hosts, and POE::Component::Resolver will not be involved. You should see different behavior for domains that aren't in /etc/hosts. After fixing 76550, your test program for 76987 does exit after POE::Component::Resolver's idle_timeout expires. Line numbers are different because I've been reducing your test program. I am attaching my version. % perl -Ilib t/rt76987.t Sat May 5 19:21:37 2012 408 Request Timeout at t/rt76987.t line 63. Sat May 5 19:21:47 2012 1 at t/rt76987.t line 53.
Subject: rt76987.t
#!/usr/bin/perl -w use utf8; use warnings; use strict; use POE; use POE::Component::Client::HTTP; use POE::Component::Client::Keepalive; use HTTP::Request::Common qw(GET); $SIG{__WARN__} = sub { my $str = shift; warn scalar(localtime) . " $str"; }; my $pool = POE::Component::Client::Keepalive->new( keep_alive => 3, max_open => 2, max_per_host => 2, timeout => 7, resolver => POE::Component::Resolver->new( max_resolvers => 1, idle_timeout => 10, ), ); POE::Component::Client::HTTP->spawn( Alias => 'ua', FollowRedirects => 3, Timeout => 0.001, MaxSize => 1024, Agent => '', ConnectionManager => $pool, Streaming => 0, ); POE::Session->create( inline_states => { _start => sub { my $url = "http://localhost/"; my $request = HTTP::Request->new('GET', $url); $poe_kernel->post("ua" => "request", "response", $request); }, _stop => sub { }, response => \&response, }, ); POE::Kernel->run(); warn 1; exit; sub response { my ($request_packet, $response_packet) = @_[ARG0, ARG1]; my $http_request = $request_packet->[0]; my $tag = $request_packet->[1]; my $http_response = $response_packet->[0]; warn $http_response->status_line; }
From: kozunov [...] gmail.com
Суб Май 05 19:23:10 2012, RCAPUTO писал: Show quoted text
> I think this is probably the same as ticket 76550. The "localhost" > domain will be resolved > through /etc/hosts, and POE::Component::Resolver will not be involved. > You should see > different behavior for domains that aren't in /etc/hosts. > > After fixing 76550, your test program for 76987 does exit after > POE::Component::Resolver's > idle_timeout expires. Line numbers are different because I've been > reducing your test program. > I am attaching my version. > > % perl -Ilib t/rt76987.t > Sat May 5 19:21:37 2012 408 Request Timeout at t/rt76987.t line 63. > Sat May 5 19:21:47 2012 1 at t/rt76987.t line 53.
I think it's different problem. I try to explain it. Problem occur when resolver answer very long. I emulate delay from dns in my example just add sleep(25) for the first request. In this case you must wait until resolver answer(25 second) and after that you must wait POE::Component::Resolver's idle_timeout expires(11 seconds). I don't want to wait answer from resolver when POE::Component::Client::HTTP timeout expire. Also we receive 408(Request Timeout) for all domains because POE::Component::Client::HTTP timeout expire before resolver send answer.
Subject: for_cpan3.pl
#!/usr/bin/perl use utf8; use strict; use warnings; use Data::Dumper; use POE 1.350; use POE::Component::Client::HTTP 0.944; use POE::Component::Client::Keepalive 0.270; use HTTP::Request::Common qw(GET HEAD); use HTTP::Cookies; use ResolverDelay; $SIG{__WARN__} = sub { my $str = shift; warn scalar(localtime)." $str"; }; open(my $fh, '>', $ResolverDelay::file) or die "Can't open file: $!"; print $fh 0; close $fh; my @domains = ('yandex.com', 'google.com', 'gmail.com', 'cnn.com', 'bing.com', 'www.bbc.co.uk'); my $pool = POE::Component::Client::Keepalive->new( keep_alive => 3, max_open => 2, max_per_host => 2, timeout => 7, resolver => POE::Component::Resolver->new( max_resolvers => 1, idle_timeout => 11, sidecar_program => [ '/usr/bin/perl', (map { "-I$_" } @INC), '-MResolverDelay', '-e', 'ResolverDelay::main()' ] ), ); POE::Component::Client::HTTP->spawn( Alias => 'ua', FollowRedirects => 3, Timeout => 3, Agent => '', ConnectionManager => $pool, Streaming => 0, CookieJar => HTTP::Cookies->new(), ); for (1..3) { POE::Session->create( inline_states => { _start => sub { my $kernel = $_[KERNEL]; $kernel->yield('next_domain'); }, _stop => sub {}, response => \&response, next_domain => \&next_domain, }, ); } POE::Kernel->run(); warn 1; exit; sub response { my ($kernel, $request_packet, $response_packet) = @_[KERNEL, ARG0, ARG1]; my $http_request = $request_packet->[0]; my $tag = $request_packet->[1]; my $http_response = $response_packet->[0]; warn $http_response->status_line, " $tag->{domain}"; # warn Dumper $http_response; $kernel->yield('next_domain'); } sub next_domain { my $kernel = $_[KERNEL]; if (@domains) { my $domain = shift @domains; my $header = [TE => 'chunked,identity', Connection => "close"]; my $url = "http://$domain/"; my $request = HTTP::Request->new( 'GET', $url, $header); $kernel->post( "ua" => "request", "response", $request, {domain=>$domain}); } }
Subject: ResolverDelay.pm
package ResolverDelay; use utf8; use strict; use warnings; use Storable qw(nfreeze thaw); use Socket::GetAddrInfo qw(:newapi getaddrinfo); use POSIX qw(:fcntl_h); our $file = '/tmp/for_cpan'; my ($count, $fh); if (-e $file) { open($fh, "+<", $file) or die "Can't open file $file: $!"; $count = <$fh>; } sub main { my $buffer = ""; my $read_length; binmode(STDIN); binmode(STDOUT); select STDOUT; $| = 1; use bytes; unless ($count) { $count++; seek($fh, 0, SEEK_SET); print $fh $count; close $fh; sleep(25); } while (1) { if (defined $read_length) { if (length($buffer) >= $read_length) { my $request = thaw(substr($buffer, 0, $read_length, "")); $read_length = undef; my ($request_id, $host, $service, $hints) = @$request; my ($err, @addrs) = getaddrinfo($host, $service, $hints); my $streamable = nfreeze( [ $request_id, $err, \@addrs ] ); my $stream = length($streamable) . chr(0) . $streamable; my $octets_wrote = syswrite(STDOUT, $stream); die $! unless $octets_wrote == length($stream); next; } } elsif ($buffer =~ s/^(\d+)\0//) { $read_length = $1; next; } my $octets_read = sysread(STDIN, $buffer, 4096, length($buffer)); last unless $octets_read; } exit 0; } 1;
On Thu May 10 14:21:26 2012, sergei wrote: Show quoted text
> > I think it's different problem. > I try to explain it. > Problem occur when resolver answer very long. > I emulate delay from dns in my example just add sleep(25) for the first > request. > > In this case you must wait until resolver answer(25 second) and after > that you must wait POE::Component::Resolver's idle_timeout expires(11 > seconds). I don't want to wait answer from resolver when > POE::Component::Client::HTTP timeout expire. > > Also we receive 408(Request Timeout) for all domains because > POE::Component::Client::HTTP timeout expire > before resolver send answer.
I think I understand the problem, and I have changed POE::Component::Resolver and POE::Component::Client::Keepalive to resolve it. Your test program produces this output after my changes. Is this good? Sun May 13 04:35:57 2012 posting request for domain yandex.com at rt76987.t line 91, <$fh> line 1. Sun May 13 04:35:57 2012 posting request for domain google.com at rt76987.t line 91, <$fh> line 1. Sun May 13 04:35:57 2012 posting request for domain gmail.com at rt76987.t line 91, <$fh> line 1. (3s HTTP timeout occurs.) (All three requests time out.) (First ResolverDelay process is killed, and a new one is started.) Sun May 13 04:36:00 2012 408 Request Timeout yandex.com at rt76987.t line 78. Sun May 13 04:36:00 2012 posting request for domain cnn.com at rt76987.t line 91. Sun May 13 04:36:00 2012 408 Request Timeout google.com at rt76987.t line 78. Sun May 13 04:36:00 2012 408 Request Timeout gmail.com at rt76987.t line 78. Sun May 13 04:36:00 2012 posting request for domain bing.com at rt76987.t line 91. Sun May 13 04:36:00 2012 posting request for domain www.bbc.co.uk at rt76987.t line 91. Sun May 13 04:36:02 2012 200 OK cnn.com at rt76987.t line 78. Sun May 13 04:36:02 2012 200 OK bing.com at rt76987.t line 78. Sun May 13 04:36:02 2012 200 OK www.bbc.co.uk at rt76987.t line 78. (11sec POE::Component::Resolver idle_timeout occurs.) Sun May 13 04:36:12 2012 1 at rt76987.t line 68.
I've just released the following modules, which you should update to: POE::Component::Resolver v0.917 POE::Component::Client::Keepalive v0.271 POE::Component::Client::HTTP v0.946 These modules should resolve this ticket and a few others. Your bug reports and test programs have been very helpful. Thank you again.