Skip Menu |

This queue is for tickets about the libwww-perl CPAN distribution.

Report information
The Basics
Id: 32719
Status: resolved
Priority: 0/
Queue: libwww-perl

People
Owner: Nobody in particular
Requestors: aksuska [...] esoteritech.com
Cc:
AdminCc:

Bug Information
Severity: Critical
Broken in: 6.04
Fixed in: (no value)



Subject: LWP 5.808 Bug (or Feature Request)
Date: Sun, 27 Jan 2008 19:13:40 -0700
To: <bug-libwww-perl [...] rt.cpan.org>
From: Keary Suska <aksuska [...] esoteritech.com>
I have been having problems with LWP, but having to do with servers with non-standard chunked transfer encoding--speciically when the content length line is not provided. Browsers don't seem to have issues with this, and (IMHO) I think LWP shouldn't either. I modified Methods.pm to get my installation to work. It's probably kludgey, and may not work in all instances, but here it is if it helps: *** /usr/lib/perl5/site_perl/5.8.0/Net/HTTP/Methods.pm.bak 2005-12-07 03:01:37.000000000 -0700 --- /usr/lib/perl5/site_perl/5.8.0/Net/HTTP/Methods.pm 2008-01-27 18:58:34.000000000 -0700 *************** *** 434,440 **** if !defined($line) || $line ne ""; $line = my_readline($self); } ! die "EOF when chunk header expected" unless defined($line); my $chunk_len = $line; $chunk_len =~ s/;.*//; # ignore potential chunk parameters unless ($chunk_len =~ /^([\da-fA-F]+)\s*$/) { --- 434,448 ---- if !defined($line) || $line ne ""; $line = my_readline($self); } ! ! #die "EOF when chunk header expected" unless defined($line); ! if( ! defined($line) ) { ! delete ${*$self}{'http_chunked'}; # say not chunked ! # read until eof ! $size ||= 8*1024; ! return my_read($self, $$buf_ref, $size); ! } ! my $chunk_len = $line; $chunk_len =~ s/;.*//; # ignore potential chunk parameters unless ($chunk_len =~ /^([\da-fA-F]+)\s*$/) { Thanks, Keary Suska Esoteritech, Inc.
Do you have examples of sites or servers that need this treatment? If the server promise to return a chunked reply I think it better do it.
Subject: PATCH: Laxed transfer-encoding chunked (confirmed)
From: MARKSTOS [...] cpan.org
On Sun Jan 27 21:14:02 2008, aksuska@esoteritech.com wrote: Show quoted text
> I have been having problems with LWP, but having to do with servers with > non-standard chunked transfer encoding--speciically when the content
length Show quoted text
> line is not provided. Browsers don't seem to have issues with this, and > (IMHO) I think LWP shouldn't either. > > I modified Methods.pm to get my installation to work. It's probably
kludgey, Show quoted text
> and may not work in all instances, but here it is if it helps:
I can confirm both that this issue is real, and that this patch addresses it. I'm attaching an updated version of the patch which applies cleanly for me to the latest release. There are only formatting changes, no logic changes. Here's how I triggered it: Inside of a Perl application running under mod_perl Apache::Registry, I wanted to return a true 404 response. I used CGI.pm to set -status => 404 With my Apache 1.3x, that successfully returns my 404 ErrorDoc page with a 404 status code. (Yay!). The result works fine in Firefox and Konqueror, but LWP returns a 500 code. If LWP is forced to send a HTTP/1.0 request, that seems to workaround it, although I wasn't able to get it to work through Test::WWW::Mechanize. When it sends a HTTP/1.1 request, it seems the end of the chunk isn't declared by the server. There some other reports out there of people experiencing this error, sometimes reporting that it happens with LWP but not normal browsers: http://www.google.com/search?hl=en&safe=off&q=500+%22EOF+when+chunk+header+expected%22++LWP+browser+chunked&btnG=Search There's a possibility I'll launch this code to a production website soon, and then I'll have a URL you can test with.
--- blib/lib/Net/HTTP/Methods.pm.orig Wed Apr 16 06:01:49 2008 +++ blib/lib/Net/HTTP/Methods.pm Thu May 1 11:15:08 2008 @@ -463,7 +463,15 @@ if !defined($line) || $line ne ""; $line = my_readline($self); } - die "EOF when chunk header expected" unless defined($line); + + #die "EOF when chunk header expected" unless defined($line); + if(not defined $line) { + delete ${*$self}{'http_chunked'}; # say not chunked + # read until eof + $size ||= 8*1024; + return my_read($self, $$buf_ref, $size); + } + my $chunk_len = $line; $chunk_len =~ s/;.*//; # ignore potential chunk parameters unless ($chunk_len =~ /^([\da-fA-F]+)\s*$/) {
From: blum [...] pobox.com
I also stumbled upon this problem. Popular browsers worked, as well as curl, but not LWP. I've had two instances of this problem, and in both cases the page was and ASP that returned a file on-demand. I suppose either the programmers of those pages did something wrong, or ASP+IIS aren't able to handle it well. A "laxed" approach to the problem is necessary IMHO, or at least a configuration option to toggle it, cause such errors exist in the wild. The patch provided above worked perfectly, thanks very much.
To make this patch easier to apply, I've made it available via git as well: http://github.com/markstos/libwww- perl/commit/021f2305e51f1eae2f7a911cbf858e8b0f5dcb68 Mark
I experience a very similar problem using libwww-perl-6.04 when trying to retrieve the following URL: https://online.westpac.com.au/esis/Login/SrvPage A transcript of my lwp-request session is attached, with specific excerpts below: $ lwp-request -Ed 'https://online.westpac.com.au/esis/Login/SrvPage' GET https://online.westpac.com.au/esis/Login/SrvPage User-Agent: lwp-request/6.03 libwww-perl/6.04 200 OK ... Client-Aborted: die ... Client-Transfer-Encoding: chunked X-Died: Missing newline after chunk data: 'Terms and Conditions</a>' at [snip]/Net/HTTP/Methods.pm line 481. ...
Subject: lwp-request.txt
$ lwp-request -v This is lwp-request version 6.03 (libwww-perl-6.04) Copyright 1995-1999, Gisle Aas. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. $ lwp-request -Ed 'https://online.westpac.com.au/esis/Login/SrvPage' GET https://online.westpac.com.au/esis/Login/SrvPage User-Agent: lwp-request/6.03 libwww-perl/6.04 200 OK Cache-Control: no-cache Date: Thu, 22 Mar 2012 11:59:56 GMT Pragma: No-cache Content-Language: en-US Content-Type: text/html;charset=ISO-8859-1 Expires: Thu, 01 Jan 1970 00:00:00 GMT Client-Aborted: die Client-Date: Thu, 22 Mar 2012 11:59:56 GMT Client-Peer: 203.24.6.136:443 Client-Response-Num: 1 Client-SSL-Cert-Issuer: /C=US/O=VeriSign, Inc./OU=VeriSign Trust Network/OU=Terms of use at https://www.verisign.com/rpa (c)06/CN=VeriSign Class 3 Extended Validation SSL SGC CA Client-SSL-Cert-Subject: /1.3.6.1.4.1.311.60.2.1.3=AU/1.3.6.1.4.1.311.60.2.1.2=./1.3.6.1.4.1.311.60.2.1.1=./businessCategory=Private Organization/serialNumber=007 457 141/C=AU/postalCode=2000/ST=New South Wales/L=SYDNEY/street=L 20 275 KENT ST/O=Westpac Banking Corporation/OU=IOLB/CN=online.westpac.com.au Client-SSL-Cipher: AES256-SHA Client-SSL-Socket-Class: IO::Socket::SSL Client-Transfer-Encoding: chunked Link: </esis/furniture/v2.7/ESI/css/_esi.css>; id="css-esi-default"; media="screen,print"; rel="stylesheet"; type="text/css" Link: </esis/furniture/v2.7/ESI/css/esi_iphone.css>; media="only screen and (max-device-width: 480px)"; rel="stylesheet"; type="text/css" Link: </esis/furniture/v2.7/css/print.css>; id="css-print"; media="print"; rel="stylesheet"; type="text/css" Link: </esis/furniture/v2.7/images/favicon.ico>; rel="shortcut icon"; type="image/x-icon" Link: <http://www.westpac.com.au/apple-touch-icon.png>; rel="apple-touch-icon" P3p: CP="NON CUR OTPi OUR NOR UNI" Set-Cookie: PLAB=wbc; Domain=online.westpac.com.au; Path=/; Secure Set-Cookie: esisapp=wol; Domain=online.westpac.com.au; Path=/; Secure Set-Cookie: WAS6=f4e7b083; Version="1"; Path=/ Title: Sign in to Westpac Online Banking - Westpac X-Died: Missing newline after chunk data: 'Terms and Conditions</a>' at /home/lx/perl5/perlbrew/perls/perl-5.12.3/lib/site_perl/5.12.3/Net/HTTP/Methods.pm line 481. X-Meta-Description: Sign in to Westpac Online Banking. X-Meta-Keywords: Online Banking, Internet Banking, Westpac Online, Sign in, Log in, X-Meta-Robots: noindex, nofollow X-Meta-Viewport: width=960, user-scalable=yes X-Meta-Westpac:Pagename: Westpac Online Banking
On Thu Mar 22 23:53:52 2012, LXP wrote: Show quoted text
> I experience a very similar problem using libwww-perl-6.04 when > trying to retrieve the following URL: > > https://online.westpac.com.au/esis/Login/SrvPage
I have been able to work around this problem by forcing HTTP 1.0 requests in this manner (also attached): use LWP::Protocol::https10 (); use Scope::Guard (); use WWW::Mechanize (); my $m = WWW::Mechanize->new( ... ); { my $curr_implementor = LWP::Protocol::implementor('https'); # Ensure that the HTTPS implementor is not changed for # other LWP::UserAgent instances. my $reverter = sub { LWP::Protocol::implementor('https', $curr_implementor); }; LWP::Protocol::implementor('https', 'LWP::Protocol::https10'); my $guard = Scope::Guard->new($reverter); $m->get( ... ); } Because changing the implementor for a protocol is process- global, Scope::Guard is used to localise this change. LWP::Protocol::http10 is described by its documentation as deprecated; I assume that the same is true for ::https10. As use of this implementor is the only apparent way at this time to enable communication with servers that do not seem to correctly handle chunked transfer encoding, I hope that its removal can be deferred.
Subject: localised-implementor.pl
use LWP::Protocol::https10 (); use Scope::Guard (); use WWW::Mechanize (); my $m = WWW::Mechanize->new( ... ); { my $curr_implementor = LWP::Protocol::implementor('https'); # Ensure that the HTTPS implementor is not changed for # other LWP::UserAgent instances. my $reverter = sub { LWP::Protocol::implementor('https', $curr_implementor); }; LWP::Protocol::implementor('https', 'LWP::Protocol::https10'); my $guard = Scope::Guard->new($reverter); $m->get( ... ); }
We have the same problem (X-Died: newline missing after chunk data) accessing a IBM WebSphere DataPower SOA Appliance's XML management interface (with SOAP::Lite). Other useragents are not that picky. Please consider relaxing the chunk TE checks - the current implementation (combined with implementation errors on the server side) disqualifies perl for one of it's main areas: admin automation.
From: blue [...] thisisnotmyrealemail.com
On Thu May 10 07:56:05 2012, MKUTTER wrote: Show quoted text
> We have the same problem (X-Died: newline missing after chunk data) > accessing a IBM WebSphere DataPower SOA Appliance's XML management > interface (with SOAP::Lite). > > Other useragents are not that picky. > > Please consider relaxing the chunk TE checks - the current > implementation (combined with implementation errors on the server side) > disqualifies perl for one of it's main areas: admin automation.
+1. It is very inconvenient to have to work around this bug.