Bug #6099 for libwww-perl: Net::HTTP::Methods does not recognize compressed responses (on some servers)

Thu Apr 22 09:03:18 2004 Guest - Ticket created

Subject:

Net::HTTP::Methods does not recognize compressed responses (on some servers)

Net:HTTP if Compress::Zlib available add TE: deflate,gzip q=0.3 Connection: TE, close but can't handle compressed responses in some cases: it exptects headers like: Transfer-Encoding: gzip Transfer-Encoding: chunked but some server send Content-Encoding header instead of first: Content-Encoding: gzip Transfer-Encoding: chunked I've attatched patch for this simple problem. Please fix, this is important for me.

--- Methods.pm.orig 2003-12-24 01:26:53.000000000 +0200 +++ Methods.pm 2004-04-22 15:54:50.000000000 +0300 @@ -330,7 +330,7 @@ my $content_length; for (my $i = 0; $i < @headers; $i += 2) { my $h = lc($headers[$i]); - if ($h eq 'transfer-encoding') { + if ($h eq 'transfer-encoding' || $h eq 'content-encoding') { push(@te, $headers[$i+1]); } elsif ($h eq 'content-length') {

Thu Apr 22 10:43:12 2004 GAAS [...] cpan.org - Correspondence added

Content-Encoding is a totally different concept. I don't believe that Net::HTTP should care about it at all, and this patch is certainly wrong. There have been discussions about making LWP have an option to automatically decompress content. Are you using LWP or Net::HTTP directly?

Thu Apr 22 17:31:05 2004 Guest - Correspondence added

From:

tech [...] askold.net

[GAAS - Thu Apr 22 10:43:12 2004]: I'm using Net::HTTP indirectly, via LWP. If this patch is not acceptable, is it possible to add an option for LWP, to tell Net::HTTP do not send "TE: deflate,gzip;q=0.3" header (at least, as workaround, do not provoke server to send compressed data) This is the dump of HTTP request. Net::HTTP can't handle it ... (it returns compressed data as result on $response->content) GET /some-url HTTP/1.1 TE: deflate,gzip;q=0.3 Connection: TE, close Host: www.linkator.ru User-Agent: Mozilla/4.0 (compatible; MSIE 5.5; Windows 98) HTTP/1.1 200 OK Date: Thu, 22 Apr 2004 21:23:52 GMT Server: CoffeeMaker/1.1 (Unix) Cache-Control: max-age=0 Expires: Thu, 22 Apr 2004 21:23:52 GMT Last-Modified: Wed, 21 Apr 2004 01:23:52 GMT Content-Encoding: gzip Set-Cookie: key=a958afaeac0c10d23b2cb63336546cc2; expires=Thu, 22-Apr-2004 22:23:53 GMT Connection: close Transfer-Encoding: chunked Content-Type: text/html; charset=windows-1251 541 [chunk of gzipped data] Show quoted text

> Content-Encoding is a totally different concept. I > don't believe that Net::HTTP should care about it at all, > and this patch is certainly wrong. > > There have been discussions about making LWP have an option > to automatically decompress content. Are you using LWP or > Net::HTTP directly?

Thu Apr 22 17:35:14 2004 Guest - Correspondence added

[guest - Thu Apr 22 17:31:05 2004]: Understood, you mean that LWP (not Net::HTTP) must look for 'Content-Encoding' header and decompress data if such header found. Is this feature planned in LWP ?

Fri Apr 23 04:34:50 2004 GAAS [...] cpan.org - Correspondence added

[guest - Thu Apr 22 17:35:14 2004]: Show quoted text

> Is this feature planned in LWP ?

Yes, but it is yet not totally clear how it will work. It is more likely that WWW:Mechanize gets it first.

Fri Apr 23 04:38:38 2004 GAAS [...] cpan.org - Correspondence added

[guest - Thu Apr 22 17:31:05 2004]: Show quoted text

> If this patch is not acceptable, is it possible to add an option > for LWP, to tell Net::HTTP do not send "TE: deflate,gzip;q=0.3" header

There is already the SendTE option for Net::HTTP. See <http://www.mail-archive.com/libwww@perl.org/msg04758.html>.

Fri Apr 23 04:50:23 2004 Guest - Correspondence added

[GAAS - Fri Apr 23 04:38:38 2004]: I know that there is send_te in Net::HTTP, but there is no way to tell it when i'm using Net::HTTP indirectly - via LWP. Thank you for your answers. Show quoted text

> [guest - Thu Apr 22 17:31:05 2004]: >

> > If this patch is not acceptable, is it possible to add an option > > for LWP, to tell Net::HTTP do not send "TE: deflate,gzip;q=0.3" header

> > There is already the SendTE option for Net::HTTP. See > <http://www.mail-archive.com/libwww@perl.org/msg04758.html>.

Fri Apr 23 04:51:59 2004 Guest - Correspondence added

[guest - Fri Apr 23 04:50:23 2004]: Sorry, never mind about this post .... I must read more carefully. Thank you ! Show quoted text

> [GAAS - Fri Apr 23 04:38:38 2004]: > > I know that there is send_te in Net::HTTP, but there is no way > to tell it when i'm using Net::HTTP indirectly - via LWP. > > Thank you for your answers. >

> > [guest - Thu Apr 22 17:31:05 2004]: > >

> > > If this patch is not acceptable, is it possible to add an option > > > for LWP, to tell Net::HTTP do not send "TE: deflate,gzip;q=0.3" header

> > > > There is already the SendTE option for Net::HTTP. See > > <http://www.mail-archive.com/libwww@perl.org/msg04758.html>.

> >

Wed Jun 16 05:05:12 2004 GAAS [...] cpan.org - Status changed from 'new' to 'resolved'