Bug #74469 for IO-Socket-SSL: infinite looping?

Sat Jan 28 02:57:58 2012 vindex [...] apartia.org - Ticket created

Subject:

infinite looping?

I have a web scraping script that uses https hosts and it hangs after a while. A "perl -d:Trace script" gives me this output after a while and it seems to loop until I CTRL-C it. Please see attached output.

Subject:

perl_error.txt

Message body is not shown because it is too large.

Mon Jan 30 08:01:15 2012 Steffen_Ullrich [...] genua.de - Correspondence added

Am Sa 28. Jan 2012, 02:57:58, vindex schrieb: Show quoted text

> I have a web scraping script that uses https hosts and it hangs after a > while. A "perl -d:Trace script" gives me this output after a while and > it seems to loop until I CTRL-C it. > > Please see attached output.

Hi, Given the output it looks like Net::HTTP::Methods does a kind of busy waiting on a non-blocking socket (wait 0.1 seconds, then try to read again) while trying to read data, which the peer does not send. It looks like, that there is simply a server, which keeps the connection open while not sending any data. This might be a problem with the server or an error in your script or Net::HTTP::Methods or whatever, but has nothing to do with IO::Socket::SSL. Therefore I reject the bug. Regards, Steffen

Mon Jan 30 08:01:17 2012 The RT System itself - Status changed from 'new' to 'open'

Mon Jan 30 08:01:18 2012 Steffen_Ullrich [...] genua.de - Status changed from 'open' to 'rejected'

Mon Jan 30 08:04:34 2012 vindex [...] apartia.org - Correspondence added

Subject:	Re: [rt.cpan.org #74469] infinite looping?
Date:	Mon, 30 Jan 2012 14:04:17 +0100
To:	Steffen Ullrich via RT <bug-IO-Socket-SSL [...] rt.cpan.org>
From:	Louis-David Mitterrand <vindex [...] apartia.org>

On Mon, Jan 30, 2012 at 08:01:16AM -0500, Steffen Ullrich via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=74469 > > > Am Sa 28. Jan 2012, 02:57:58, vindex schrieb:

> > I have a web scraping script that uses https hosts and it hangs after a > > while. A "perl -d:Trace script" gives me this output after a while and > > it seems to loop until I CTRL-C it. > > > > Please see attached output.

> > > Hi, > Given the output it looks like Net::HTTP::Methods does a kind of busy > waiting on a non-blocking socket (wait 0.1 seconds, then try > to read again) while trying to read data, which the peer does not send. > It looks like, that there is simply a server, which keeps the connection > open while not sending any data. > This might be a problem with the server or an error in your script or > Net::HTTP::Methods or whatever, but has nothing to do with > IO::Socket::SSL. > Therefore I reject the bug.

Isn't it a known issue of IO::Socket::SSL not implementing LWP::UserAgent's 'timeout' parameter? (whereas Net::SSL does)

Mon Jan 30 08:04:35 2012 The RT System itself - Status changed from 'rejected' to 'open'

Mon Jan 30 08:05:32 2012 vindex [...] apartia.org - Correspondence added

Subject:	Re: [rt.cpan.org #74469] infinite looping?
Date:	Mon, 30 Jan 2012 14:05:24 +0100
To:	Steffen Ullrich via RT <bug-IO-Socket-SSL [...] rt.cpan.org>
From:	Louis-David Mitterrand <vindex [...] apartia.org>

On Mon, Jan 30, 2012 at 02:04:17PM +0100, Julius Vindex wrote: Show quoted text

> On Mon, Jan 30, 2012 at 08:01:16AM -0500, Steffen Ullrich via RT wrote:

> > <URL: https://rt.cpan.org/Ticket/Display.html?id=74469 > > > > > Am Sa 28. Jan 2012, 02:57:58, vindex schrieb:

> > > I have a web scraping script that uses https hosts and it hangs after a > > > while. A "perl -d:Trace script" gives me this output after a while and > > > it seems to loop until I CTRL-C it. > > > > > > Please see attached output.

> > > > > > Hi, > > Given the output it looks like Net::HTTP::Methods does a kind of busy > > waiting on a non-blocking socket (wait 0.1 seconds, then try > > to read again) while trying to read data, which the peer does not send. > > It looks like, that there is simply a server, which keeps the connection > > open while not sending any data. > > This might be a problem with the server or an error in your script or > > Net::HTTP::Methods or whatever, but has nothing to do with > > IO::Socket::SSL. > > Therefore I reject the bug.

> > Isn't it a known issue of IO::Socket::SSL not implementing > LWP::UserAgent's 'timeout' parameter? (whereas Net::SSL does)

Just found it. I have exactly the same problem as this bug: https://bugzilla.redhat.com/show_bug.cgi?id=750793

Mon Jan 30 09:08:22 2012 Steffen_Ullrich [...] genua.de - Correspondence added

Show quoted text

> > I have exactly the same problem as this bug: > > https://bugzilla.redhat.com/show_bug.cgi?id=750793

Thanks for the pointer. And like the bug says - it's a problem with Net::HTTP. The attached patch should fix the issue in Net::HTTP, hope this helps for now. I'll also send it to the Net::HTTP maintainer. One reason for the whole problem is the insufficient documentation of the Timeout parameter in IO::Socket. Some people expect it to be used for read and write too, but it's only used for accept and connect. Same behavior with IO::Socket::SSL. But I think Net::SSL honors Timeout even for reads and writes.

Subject:

Methods.pm.patch

--- /usr/share/perl5/Net/HTTP/Methods.pm 2011-03-20 12:36:18.000000000 +0100 +++ Net/HTTP/Methods.pm 2012-01-30 14:50:38.556611428 +0100 @@ -242,6 +242,7 @@ sub my_readline { my $self = shift; my $what = shift; + my $timeout = $self->timeout; for (${*$self}{'http_buf'}) { my $max_line_length = ${*$self}{'http_max_line_length'}; my $pos; @@ -255,6 +256,9 @@ # need to read more data to find a line ending READ: { + if ( my $to = ${*$self}{io_socket_timeout} ) { + die "read timeout" if ! $self->can_read($to); + } my $n = $self->sysread($_, 1024, length); unless (defined $n) { redo READ if $!{EINTR};

Mon Feb 20 02:05:57 2012 Steffen_Ullrich [...] genua.de - Correspondence added

Bug closed. It's not a problem of IO::Socket::SSL. See https://rt.cpan.org/Ticket/Display.html?id=72676 for the real solution.

Mon Feb 20 02:05:58 2012 Steffen_Ullrich [...] genua.de - Status changed from 'open' to 'rejected'