Bug #44002 for libwww-perl: LWP::UserAgent doesn't follow "location" (lower-case "Location") headers

Tue Mar 10 09:28:49 2009 avar [...] cpan.org - Ticket created

Subject:

LWP::UserAgent doesn't follow "location" (lower-case "Location") headers

When using LWP::UserAgent to interface with imageshack.us the library stopped following location: header redirects since it only looks for the upper-case Location: header (see attached file for example response). This is probably RFC incompliant behavior violating section "14.30 Location" of RFC 2616, nevertheless Firefox -- and I presume all other major browsers as imageshack is a major website -- will follow the lower-case form. The workaround is to do something like the following in the program using LWP::UserAgent: # Imageshack sets a "location" (not a "Location") header my $location = $res->header("location"); my $res2 = $ua->get($location); But perhaps libwww-perl should support this abuse of the HTTP spec?

Subject:

location-header.txt

HTTP/1.1 302 Found X-Powered-By: PHP/5.2.6 Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Set-Cookie: latest=img60; expires=Fri, 05-Mar-2010 13:12:15 GMT; path=/; domain=.imageshack.us Set-Cookie: always_opt=-1; path=/; domain=.imageshack.us Set-Cookie: rem_bar=-1; path=/; domain=.imageshack.us location: http://img60.imageshack.us/content.php?page=done&l=img60/7083/img9732.jpg Content-type: text/html Transfer-Encoding: chunked Connection: close Date: Tue, 10 Mar 2009 13:12:20 GMT Server: lighttpd/1.5.0 1 0

Tue Mar 10 09:48:26 2009 avar [...] cpan.org - Correspondence added

On Tue Mar 10 09:28:49 2009, AVAR wrote: Show quoted text

> This is probably RFC incompliant behavior violating section "14.30 > Location" of RFC 2616, nevertheless Firefox -- and I presume all other > major browsers as imageshack is a major website -- will follow the > lower-case form.

Actually this is perfectly compliant with RFC 2616, "Field names are case-insensitive." according to section 4.2 of the spec. Which means this is probably a more general problem in libwww-perl as it uses camel-cased version of header names in numerous ->header($str) calls in its code.

Tue Mar 10 09:48:27 2009 avar [...] cpan.org - Status changed from 'new' to 'open'

Tue Mar 10 10:06:19 2009 GAAS [...] cpan.org - Correspondence added

I'm quite sure that LWP doesn't care if the header field names are all lower case. Something else must explain the missing redirect. Is the server you are communicating with public? If so, can you produce a small test case to demonstrate the issue?

Tue Mar 10 11:58:24 2009 avar [...] cpan.org - Correspondence added

On Tue Mar 10 10:06:19 2009, GAAS wrote: Show quoted text

> I'm quite sure that LWP doesn't care if the header field names are all > lower case. Something else > must explain the missing redirect.

You're right. Further debugging reveals that $res->header($str) doesn't care about the casing of $str. The problem was that the webserver in question was redirecting from a POST request which LWP::UserAgent can optionally support with: my $ua = LWP::UserAgent->new(requests_redirectable => [ qw<HEAD GET Show quoted text

POST> ]);

For reference I'm attaching the script I used to debug this, usage: perl mirror.pl [hack|no-hack] URL Where URL is a link to an image, e.g. a JPG.

#!/usr/bin/env perl use strict; use warnings; use LWP::UserAgent; my $location_hack = $ARGV[0] eq 'hack'; print mirror_imageshack($ARGV[1]), "\n"; sub mirror_imageshack { my ($chan_image) = @_; my $ua = LWP::UserAgent->new( cookie_jar => {}, requests_redirectable => [ qw<HEAD GET POST> ], ); $ua->get( "http://imageshack.us/" ); $ua->default_header('Referer' => "http://imageshack.us/"); my $res = $ua->post( "http://www.imageshack.us/transload.php", Content_Type => 'multipart/form-data', Content => [ uploadtype => 'on', url => $chan_image, email => '', MAX_FILE_SIZE => 13145728, refer => '', brand => '', optsize => 'resample' ], ); if ($location_hack) { # Imageshack sets a "location" (not a "Location") header my $location = $res->header("location"); $res = $ua->get($location); unless ($res->is_success()) { warn "Location hack request failed"; return; } } my $content = $res->content(); if ($content =~ m[Direct <a.*? href='(.*?)'>link</a> to image]) { return $1; } return; }

Mon Jun 15 12:41:18 2009 GAAS [...] cpan.org - Status changed from 'open' to 'rejected'