Skip Menu |

This queue is for tickets about the AnyEvent-HTTP-Message CPAN distribution.

Report information
The Basics
Id: 85665
Status: resolved
Priority: 0/
Queue: AnyEvent-HTTP-Message

People
Owner: Nobody in particular
Requestors: blue [...] thisisnotmyrealemail.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: problem using HTTP::Request with POST
Attached is a script that demonstrates an issue when posting with a HTTP::Request. When using regular AnyEvent::HTTP::http_request it works fine, but when using HTTP::Request and AnyEvent::HTTP::Request it results in a timeout every time. I'm not sure what the issue is, cause AnyEvent::HTTP has no introspection abilities and I currently don't have access to wireshark to sniff the traffic.
Subject: a.pl
#!/usr/bin/perl use 5.016; use warnings; use AnyEvent::HTTP; use AnyEvent::HTTP::Request; use AnyEvent::HTTP::Response; use Data::Printer; use HTTP::Request::Common; my $proxy = 'http://45bytes.info/'; my %headers = ( accept => 'text/html,application/xhtml+xml,application/xml;' . 'q=0.9,*/*;q=0.8', accept_language => 'en-US,en;q=0.5', accept_encoding => 'gzip,deflate', connection => 'keep-alive', referer => $proxy, ); my $url = "${proxy}includes/process.php?action=update"; my $cv = AE::cv; # Works http_request( POST => $url, body => 'u=http%3A%2F%2Fwww.google.com%2F', headers => { %headers, 'content-type' => "application/x-www-form-urlencoded", }, cookie_jar => {}, timeout => 30, sub { p @_ } ); my $req = POST $url, %headers, content => [ u => 'http://www.google.com/', ]; # Doesn't work. Results in timeout. AnyEvent::HTTP::Request->new($req => { params => { cookie_jar => {}, timeout => 30, }, cb => sub { p @_ } })->send; $cv->recv;
From: blue [...] thisisnotmyrealemail.com
On Mon May 27 23:10:09 2013, blue wrote: Show quoted text
> Attached is a script that demonstrates an issue when posting with a > HTTP::Request. When using regular AnyEvent::HTTP::http_request it > works fine, but when using HTTP::Request and > AnyEvent::HTTP::Request it results in a timeout every time. > I'm not sure what the issue is, cause AnyEvent::HTTP has no > introspection abilities and I currently don't have access to > wireshark to sniff the traffic.
I think I see what's happening, but not why. The post results in a redirect and when AHR is used it erroneously keeps the Content-Length header from the initial post request and applies it to the redirected get request as well. I used Charles debugging proxy to trace the requests: POST /includes/process.php?action=update HTTP/1.1 User-agent: Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.15; +http://software.schmorp.de/pkg/AnyEvent) Referer: http://45bytes.info/ Connection: close Te Accept-encoding: gzip,deflate Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-language: en-US,en;q=0.5 Content-length: 32 Host: 45bytes.info Content-type: application/x-www-form-urlencoded Te: trailers u=http%3A%2F%2Fwww.google.com%2F GET /secure.php?u=qS9oLdg0cTVBaUQ1293naF9S&b=0&f=norefer HTTP/1.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Cookie: s=dd60509944dcb4ed49a66f21b77a251e Content-type: application/x-www-form-urlencoded Te: trailers Host: 45bytes.info Accept-language: en-US,en;q=0.5 Referer: http://45bytes.info/ User-agent: Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.15; +http://software.schmorp.de/pkg/AnyEvent) Accept-encoding: gzip,deflate Connection: Te POST /includes/process.php?action=update HTTP/1.1 Accept-encoding: gzip,deflate Connection: close Te Referer: http://45bytes.info/ User-agent: Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.15; +http://software.schmorp.de/pkg/AnyEvent) Content-length: 32 Host: 45bytes.info Content-type: application/x-www-form-urlencoded Te: trailers Accept-language: en-US,en;q=0.5 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 u=http%3A%2F%2Fwww.google.com%2F ## Times out on this one GET /secure.php?u=nTtJjiBUIWybn53mZL4KcLyo&b=0&f=norefer HTTP/1.1 Accept-encoding: gzip,deflate Connection: Te Referer: http://45bytes.info/ User-agent: Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.15; +http://software.schmorp.de/pkg/AnyEvent) Te: trailers Content-type: application/x-www-form-urlencoded Content-length: 32 Host: 45bytes.info Accept-language: en-US,en;q=0.5 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Cookie: s=3450a31f7cce7bcb4da274eb74e674f8
On Mon May 27 20:10:09 2013, blue wrote: Show quoted text
> Attached is a script that demonstrates an issue when posting with a > HTTP::Request. When using regular AnyEvent::HTTP::http_request it > works fine, but when using HTTP::Request and > AnyEvent::HTTP::Request it results in a timeout every time. > I'm not sure what the issue is, cause AnyEvent::HTTP has no > introspection abilities and I currently don't have access to > wireshark to sniff the traffic.
Very strange... I changed the two calls to print the args that are passed to http_request and the only difference I saw was that Content-Length was passed (something HTTP::Request must be doing automatically). I added the content-length param to the plain http_request call and then that one also times out. I don't know why the request to that server times out when a content-length is sent, but if i post to another server it works fine. I tried commenting out the keep-alive but it still timed out. Maybe 45bytes doesn't like the content-length header? Seems odd. As for the content-length getting added automatically, that's coming from HTTP::Request. Am I missing something?
On Tue Jun 11 19:44:17 2013, RWSTAUNER wrote: Show quoted text
> On Mon May 27 20:10:09 2013, blue wrote:
> > Attached is a script that demonstrates an issue when posting with a > > HTTP::Request. When using regular AnyEvent::HTTP::http_request it > > works fine, but when using HTTP::Request and > > AnyEvent::HTTP::Request it results in a timeout every time. > > I'm not sure what the issue is, cause AnyEvent::HTTP has no > > introspection abilities and I currently don't have access to > > wireshark to sniff the traffic.
> > > Very strange... > > I changed the two calls to print the args that are passed to > http_request > and the only difference I saw was that Content-Length was passed > (something HTTP::Request must be doing automatically). > > I added the content-length param to the plain http_request call and > then that one also times out. > > I don't know why the request to that server times out when a content- > length is sent, but if i post to another server it works fine. > > I tried commenting out the keep-alive but it still timed out. > > Maybe 45bytes doesn't like the content-length header? Seems odd. > > As for the content-length getting added automatically, > that's coming from HTTP::Request. > > Am I missing something?
Looking over your second comment and thinking about what I said... The bug appears to be in the main http_request() in that it is preserving the Content-Length erroneously across the redirected GET request. Do you agree?
From: blue [...] thisisnotmyrealemail.com
On Tue Jun 11 22:44:17 2013, RWSTAUNER wrote: Show quoted text
> On Mon May 27 20:10:09 2013, blue wrote:
> > Attached is a script that demonstrates an issue when posting with a > > HTTP::Request. When using regular AnyEvent::HTTP::http_request it > > works fine, but when using HTTP::Request and > > AnyEvent::HTTP::Request it results in a timeout every time. > > I'm not sure what the issue is, cause AnyEvent::HTTP has no > > introspection abilities and I currently don't have access to > > wireshark to sniff the traffic.
> > > Very strange... > > I changed the two calls to print the args that are passed to > http_request > and the only difference I saw was that Content-Length was passed > (something HTTP::Request must be doing automatically). > > I added the content-length param to the plain http_request call and > then that one also times out. > > I don't know why the request to that server times out when a content- > length is sent, but if i post to another server it works fine. > > I tried commenting out the keep-alive but it still timed out. > > Maybe 45bytes doesn't like the content-length header? Seems odd.
Definitely a webserver issue on their end, BUT the request should fail nonetheless because: "When a Content-Length is given in a message where a message-body is allowed, its field value MUST exactly match the number of OCTETs in the message-body. HTTP/1.1 user agents MUST notify the user when an invalid length is received and detected." From: http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4 Sending a Content-Length header without any content is an error. Show quoted text
> As for the content-length getting added automatically, > that's coming from HTTP::Request.
Not sure about that. See example script using LWP::UserAgent below. Show quoted text
> > Am I missing something?
#!/usr/bin/env perl use strict; use warnings; use Data::Printer; use HTTP::Request::Common; use LWP::UserAgent; my $proxy = 'http://45bytes.info/'; my $url = "${proxy}includes/process.php?action=update"; my %headers = ( accept => 'text/html,application/xhtml+xml,application/xml;' . 'q=0.9,*/*;q=0.8', accept_language => 'en-US,en;q=0.5', accept_encoding => 'gzip,deflate', connection => 'keep-alive', referer => $proxy, ); my $ua = LWP::UserAgent->new( cookie_jar => {}, default_headers => HTTP::Headers->new(%headers), requests_redirectable => [qw( GET HEAD POST )], ); my $req = POST $url, content => [ u => 'http://www.google.com/', ]; my $res = $ua->request($req); p $res;
From: blue [...] thisisnotmyrealemail.com
On Tue Jun 11 23:08:30 2013, blue wrote: Show quoted text
> Sending a Content-Length header without any content is an error.
Correction: sending an incorrect content-length header is an error.
From: blue [...] thisisnotmyrealemail.com
On Tue Jun 11 22:47:54 2013, RWSTAUNER wrote: Show quoted text
> > Looking over your second comment and thinking about what I said... > The bug appears to be in the main http_request() in that it is > preserving the Content-Length erroneously across the redirected GET > request. > > Do you agree?
(Didn't notice this message, cause I was composing my previous response while it was sent.) If that's where the bug is, why does it only occur when AHR is used and not also when http_request is directly called?
Show quoted text
> If that's where the bug is, why does it only occur when AHR is used > and not also when http_request is directly called?
It's not fair to compare AEH to LWP because LWP operates "correctly" in that it does not erroneously send an incorrect Content-Length to the redirected GET. If you pass a content-length header to http_request it will resend that header even across the redirect GET (which is incorrect behavior). As for the "problem with the server" I wrote a small plack app to simulate the "receive POST and redirect to GET" behavior and found that it does indeed hang on a GET request with a content-length and no content... probably because GET requests can in fact have content, so I suppose if the server sees a content-length header then it's waiting for the client to send the content. (This is an idiot's guess.) So anyway, I looked into the code (of various modules) and made a few discoveries: * It is actually not HTTP::Request that adds the Content-Length header, but HTTP::Request::Common::POST() that does it. * Additionally if you use HTTP::Request (instead of HTTP::Request::Common::POST()) to build the request, LWP::Protocol::http will also set the content-length for you. * AnyEvent::HTTP passes all supplied headers blindly and overwrites the content-length header when it deems appropriate. The documentation merely states that AEH "may provide its own ...Content-Length... header..." (which can be suppressed with "undef"). It doesn't make it clear that you shouldn't provide these headers because it will pass them on inappropriately. So to me it seems that the bug is in AEH in that it sends a Content-Length header incorrectly with a new (redirect) request that has different (no) content. The documentation does not clearly state that this is the expected behavior. However, I would be glad to explicitly delete any content-length provided by HTTP::Request to work around this behavior. I appreciate the work you did in debugging because considering the discoveries mentioned above I probably wouldn't have figured it out without your investigative work. Thanks.
Show quoted text
> However, I would be glad to explicitly delete any content-length > provided by HTTP::Request to work around this behavior.
Version 0.302 has been released with the aforementioned fix. Thanks again for your help.