Bug #35018 for File-Fetch: Treat HTTP 404 Message as File with lynx

Mon Apr 14 12:21:59 2008 imacat [...] cpan.org - Ticket created

Subject:

Treat HTTP 404 Message as File with lynx

Dear Jos Boumans, Hi. This is imacat from Taiwan. I found that your File-Fetch-0.14 fetches the HTTP 404 error page as the file when lynx is used. It seems that, because lynx does not return bad code with HTTP 404, it cannot distinguish between successful or failed download and always return as success. This causes ambiguous information to the caller application, such as CPANPLUS. CPANPLUS will always gets the fake file even if the file does really not exist, and deal with it consequently. I think this is a bit serious. I would suggest you that, in _lynx_fetch(), try a "-head" request first, and process further only on 200 OK, to avoid such a problem. The terminal log is attached below. Hope that this helps. Please tell me if you need any more information, or if I could be of any help. Thank you. imacat@rinse /tmp/t % ls -a . .. imacat@rinse /tmp/t % perl -v This is perl, v5.8.8 built for x86_64-linux-thread-multi-ld Copyright 1987-2006, Larry Wall Perl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Perl 5 source kit. Complete documentation for Perl, including FAQ lists, should be found on this system using "man perl" or "perldoc perl". If you have access to the Internet, point your browser at http://www.perl.org/, the Perl Home Page. imacat@rinse /tmp/t % perl -mFile::Fetch -e'exit 1 unless File::Fetch->new(uri=>"http://localhost/nosuchfile")->fetch;' Fetch failed! HTTP response: 404 Not Found [404 Not Found] at -e line 1 Command failed: at -e line 1 imacat@rinse /tmp/t % echo $? 0 imacat@rinse /tmp/t % ls -a . .. nosuchfile imacat@rinse /tmp/t % cat nosuchfile <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /nosuchfile was not found on this server.</p> <hr> <address>Apache/2.2.6 (Unix) mod_ssl/2.2.6 OpenSSL/0.9.8g mod_bwshare/0.2.0 mod_perl/2.0.3 Perl/v5.8.8 Server at localhost Port 80</address> </body></html> imacat@rinse /tmp/t %

Wed Apr 16 22:01:57 2008 imacat [...] cpan.org - Correspondence added

Dear Jos Boumans, Hi. This is imacat from Taiwan. I made a simple patch against File-Fetch-0.14, to make it tries to make a HEAD request first and fails if not receiving an 200 OK status. Hope that this helps. Please tell me if you need any more information, or if I could be of any help. Thank you.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 diff -u -r File-Fetch-0.14.orig/lib/File/Fetch.pm File-Fetch-0.14/lib/File/Fetch.pm - --- File-Fetch-0.14.orig/lib/File/Fetch.pm 2007-12-14 20:41:01.000000000 +0800 +++ File-Fetch-0.14/lib/File/Fetch.pm 2008-04-17 09:57:33.000000000 +0800 @@ -717,6 +717,37 @@ 'lynx' )); } + ### check if the HTTP resource exists ### + if ($self->uri =~ /^https?:\/\//i) { + my $cmd = [ + $lynx, + '-head', + '-source', + "-auth=anonymous:$FROM_EMAIL", + ]; + + push @$cmd, "-connect_timeout=$TIMEOUT" if $TIMEOUT; + + ### DO NOT quote things for IPC::Run, it breaks stuff. + push @$cmd, $IPC::Cmd::USE_IPC_RUN + ? $self->uri + : QUOTE. $self->uri .QUOTE; + + + ### shell out ### + my $head; + unless(run( command => $cmd, + buffer => \$head, + verbose => $DEBUG ) + ) { + return $self->_error(loc("Command failed: %1", $head || '')); + } + + unless($head =~ /^HTTP\/\d+\.\d+ 200\b/) { + return $self->_error(loc("Command failed: %1", $head || '')); + } + } + ### write to the output file ourselves, since lynx ass_u_mes to much my $local = FileHandle->new(">$to") or return $self->_error(loc( -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkgGrroACgkQi9gubzC5S1y4/wCgpfSE/olPl8KBe01VLrApk+NZ hPUAoJpBPOcz9b08vCJmgk88rMqfoJXA =unxT -----END PGP SIGNATURE-----

Wed Apr 16 22:02:03 2008 The RT System itself - Status changed from 'new' to 'open'

Sat Apr 19 09:03:03 2008 kane [...] cpan.org - Correspondence added

On Wed Apr 16 22:01:57 2008, IMACAT wrote: Show quoted text

> Dear Jos Boumans, > > Hi. This is imacat from Taiwan. I made a simple patch against > File-Fetch-0.14, to make it tries to make a HEAD request first and fails > if not receiving an 200 OK status. Hope that this helps. Please tell > me if you need any more information, or if I could be of any help. > Thank you.

Thanks, applied

Sat Apr 19 09:03:05 2008 kane [...] cpan.org - Status changed from 'open' to 'resolved'