Bug #6487 for libwww-perl: FTP problems under LWP

Hello I am trying to retrieve a directory listing from an FTP server using LWP. I think that the site may well be sending me non-ascii characters? However, Net::FTP can handle it, and from reading LWP::Protocol::ftp.pm, there is an implication that Net::FTP is used by LWP? LWP does act as I expect if I use it to get a directory listing of the parent directory of the one I need to list. I could bypass the problem by using Net::FTP for this one ftp server, however, since the rest of my code uses LWP, I'd sooner get to the bottom of the problem. I have listed the programs here, followed by their output collected via script. I have tried to reduce the programs to their essential statements. I have changed the ftp address, username and password because I do not know if I have permission to distribute them. However, the outputs are those garnered before this change. The code is ordered: netftp.pl netftp.op lwpbad.pl lwpbad.op lwpgood.pl lwpgood.op I suspect that the fact that 'content-type' => 'application/octet-stream' is set in the HTTP::Header of the lwpbad output is wrong. It is set to 'content-type' => 'text/ftp-dir-listing' in the lwpgood output. I think that this is probably due to the fact that LWP::MediaTypes is interpreting the non-ascii character? Is it possible for me to override the assignation given. I do set the content_type of my request object to be "text/ftp-dir-listing". Bob ---------------------------------------------------------------------------------------------- #!/usr/bin/perl -w use strict; use Net::FTP; use Data::Dumper; my $ftp = Net::FTP->new("partner.abcde.ru", Debug => 0) or die "Cannot connect to partner.abcde.ru: $@"; $ftp->login("xyz",'123') or die "Cannot login ", $ftp->message; $ftp->cwd("/bases") or die "Cannot change working directory ", $ftp->message; my $dir = $ftp->dir or die "dir failed ", $ftp->message; warn "DIR is ",Dumper($dir); $ftp->quit; ---------------------------------------------------------------------------------------------- Script started on Wed Jun 2 12:25:13 2004 ]0;bob@snoogen:~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ perl netftp.pl DIR is $VAR1 = [ 'total 3870', '-rw-r--r-- 1 2133 4000 15297 May 20 12:39 advware.avc', '-rw-r--r-- 1 2133 4000 962750 May 20 12:39 avp0404.avc', '-rw-r--r-- 1 2133 4000 286326 May 20 12:39 backdoor.avc', '-rw-r--r-- 1 2133 4000 53719 May 20 12:39 ca.avc', '-rw-r--r-- 1 2133 4000 32075 Jun 2 14:54 daily.avc', '-rw-r--r-- 1 2133 4000 1843 May 20 12:39 eicar.avc', '-rw-r--r-- 1 2133 4000 19046 May 20 12:39 extr-cab.avc', '-rw-r--r-- 1 2133 4000 211150 May 20 12:39 extract.avc', '-rw-r--r-- 1 2133 4000 8401 May 20 12:39 kernel.avc', '-rw-r--r-- 1 2133 4000 5275 May 20 12:39 krndos.avc', '-rw-r--r-- 1 2133 4000 23515 May 20 12:39 krnengn.avc', '-rw-r--r-- 1 2133 4000 64482 May 20 12:39 krnexe.avc', '-rw-r--r-- 1 2133 4000 37999 May 20 12:39 krnjava.avc', '-rw-r--r-- 1 2133 4000 76929 May 20 12:39 krnmacro.avc', '-rw-r--r-- 1 2133 4000 55548 May 20 12:39 krnunp.avc', '-rw-r--r-- 1 2133 4000 141049 May 20 12:39 macro.avc', '-rw-r--r-- 1 2133 4000 14114 May 20 12:39 mail.avc', '-rw-r--r-- 1 2133 4000 115922 May 20 12:39 malware.avc', '-rw-r--r-- 1 2133 4000 375390 May 20 12:39 newexe.avc', '-rw-r--r-- 1 2133 4000 14920 May 20 12:39 newexeg.avc', '-rw-r--r-- 1 2133 4000 3162 May 20 12:39 ocr.avc', '-rw-r--r-- 1 2133 4000 27541 May 20 12:39 pornware.avc', '-rw-r--r-- 1 2133 4000 28492 May 20 12:39 riskware.avc', '-rw-r--r-- 1 2133 4000 99749 May 20 12:39 script.avc', '-rw-r--r-- 1 2133 4000 5366 May 20 12:39 smart.avc', '-rw-r--r-- 1 2133 4000 356774 May 20 12:39 trojan.avc', '-rw-r--r-- 1 2133 4000 569565 May 20 12:39 unpack.avc', '-rw-r--r-- 1 2133 4000 15450 May 20 12:39 up040402.avc', '-rw-r--r-- 1 2133 4000 18563 May 20 12:39 up040409.avc', '-rw-r--r-- 1 2133 4000 23479 May 20 12:39 up040416.avc', '-rw-r--r-- 1 2133 4000 40902 May 20 12:39 up040423.avc', '-rw-r--r-- 1 2133 4000 27378 May 20 12:39 up040430.avc', '-rw-r--r-- 1 2133 4000 19431 May 20 12:39 up040507.avc', '-rw-r--r-- 1 2133 4000 35446 May 20 12:39 up040514.avc', '-rw-r--r-- 1 2133 4000 23532 May 20 12:39 x-files.avc', '-rw-r--r-- 1 2133 4000 2040 Jun 2 14:54 xdaily.avc' ]; ]0;bob@snoogen:~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ Script done on Wed Jun 2 12:25:22 2004 ---------------------------------------------------------------------------------------------- #!/usr/bin/perl -w use strict; use LWP::UserAgent; my $url = "ftp://partner.abcde.ru/bases"; my $user = "xyz"; my $pass = "123"; my $content_type = "text/ftp-dir-listing"; my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new('GET', $url); $req->authorization_basic($user, $pass); $req->content_type($content_type); my $res = $ua->request($req); use Data::Dumper; warn "RES is ",Dumper($res); ---------------------------------------------------------------------------------------------- Script started on Wed Jun 2 12:34:13 2004 ]0;bob@snoogen:~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ perl lwpbad.pl RES is $VAR1 = bless( { '_content' => ',
.* ..,servers.lst,advware.avc,avp0404.avc,backdoor.avc³+À,ca.avcǐ, daily.avcä, eicar.avcä,extr-cab.avc³+À ,extract.avc!, kernel.avcä., krndos.avcä/,krnengn.avc0, krnexe.avcä1,krnjava.avc2,krnmacro.avc³+À3, krnunp.avcä4, macro.avcä5,mail.avc™ä6,malware.avc7, newexe.avcä8,newexeg.avc9,ocr.avc:,pornware.avc³+À<, script.avcä;,riskware.avc³+À=, smart.avcä>, trojan.avcä?, unpack.avcä@,up040402.avc³+ÀA,up040409.avc³+ÀB,up040416.avc³+ÀC,up040423.avc³+ÀD,up040430.avc³+ÀE,up040507.avc³+ÀF,up040514.avc³+ÀG,x-files.avcH,ð xdaily.avcä', '_rc' => 200, '_headers' => bless( { 'client-date' => 'Wed, 02 Jun 2004 11:34:20 GMT', 'content-type' => 'application/octet-stream', 'server' => ' ###############################################################', 'client-request-num' => 1 }, 'HTTP::Headers' ), '_msg' => 'OK', '_request' => bless( { '_content' => '', '_uri' => bless( do{\(my $o = 'ftp://partner.abcde.ru/bases')}, 'URI::ftp' ), '_headers' => bless( { 'user-agent' => 'libwww-perl/5.65', 'content-type' => 'text/ftp-dir-listing', 'authorization' => 'Basic S0FWREI6UG5RR0c4ZThXM2lKWlVTeQ==' }, 'HTTP::Headers' ), '_method' => 'GET' }, 'HTTP::Request' ) }, 'HTTP::Response' ); ]0;bob@snoogen:~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ Script done on Wed Jun 2 12:34:29 2004 ---------------------------------------------------------------------------------------------- #!/usr/bin/perl -w use strict; use LWP::UserAgent; my $url = "ftp://partner.abcde.ru/"; my $user = "xyz"; my $pass = "123"; my $content_type = "text/ftp-dir-listing"; my $ua = LWP::UserAgent->new; my $req = HTTP::Request->new('GET', $url); $req->authorization_basic($user, $pass); $req->content_type($content_type); my $res = $ua->request($req); use Data::Dumper; warn "RES is ",Dumper($res); ---------------------------------------------------------------------------------------------- Script started on Wed Jun 2 12:33:32 2004 ]0;bob@snoogen:~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ perl lwpgood.pl RES is $VAR1 = bless( { '_content' => 'total 2 drwxr-xr-x 2 2133 4000 1024 May 20 12:39 bases ', '_rc' => 200, '_headers' => bless( { 'client-date' => 'Wed, 02 Jun 2004 11:33:40 GMT', 'content-type' => 'text/ftp-dir-listing', 'content-length' => 58, 'server' => ' ###############################################################', 'client-request-num' => 1 }, 'HTTP::Headers' ), '_msg' => 'OK', '_request' => bless( { '_content' => '', '_uri' => bless( do{\(my $o = 'ftp://partner.abcde.ru/')}, 'URI::ftp' ), '_headers' => bless( { 'user-agent' => 'libwww-perl/5.65', 'content-type' => 'text/ftp-dir-listing', 'authorization' => 'Basic S0FWREI6UG5RR0c4ZThXM2lKWlVTeQ==' }, 'HTTP::Headers' ), '_method' => 'GET' }, 'HTTP::Request' ) }, 'HTTP::Response' ); ]0;bob@snoogen:~/work/accurev/VirusSignatures[bob@snoogen VirusSignatures]$ Script done on Wed Jun 2 12:33:45 2004 ----------------------------------------------------------------------------------------------