Bug #99523 for libwww-perl: Can't connect to ibs.bankwest.com.au:443

Wed Oct 15 23:01:37 2014 lxp [...] cpan.org - Ticket created

Subject:

Can't connect to ibs.bankwest.com.au:443

I am unable to access particular websites over HTTPS using libwww-perl- 6.08 from Perl v5.20.1 on an ARM-based device running Debian Wheezy. lwp-request can connect to some sites: $ perl-5.20.1/bin/lwp-request https://google.com/ | head -1 <!doctype html>... but not others: $ perl-5.20.1/bin/lwp-request https://ibs.bankwest.com.au/ Can't connect to ibs.bankwest.com.au:443 despite other things on the same system being able to: $ curl -s https://ibs.bankwest.com.au/ | head -1 <HTML><HEAD> Module versions: - LWP::Protocol::https v6.06 - IO::Socket::SSL v1.999 - Net::SSLeay v1.66 - Crypt::SSLeay v0.72 libssl-dev version: $ dpkg -s libssl-dev | grep Version: Version: 1.0.1e-2+deb7u12 Perl information: $ perl-5.20.1/bin/perl -V Summary of my perl5 (revision 5 version 20 subversion 1) configuration: Platform: osname=linux, osvers=3.7.3, archname=armv5tel-linux uname='linux plug 3.7.3 #1 preempt thu jan 17 13:00:31 mst 2013 armv5tel gnulinux ' config_args='-Dprefix=/var/lib/bankwatch/perl-5.20.1 -de -A'eval:scriptdir=/var/lib/bankwatch/perl-5.20.1/bin'' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include' ccversion='', gccversion='4.6.3', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='cc', ldflags =' -fstack-protector -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/arm-linux-gnueabi/4.6/include-fixed /usr/include/arm-linux-gnueabi /usr/lib /lib/arm-linux-gnueabi /lib /usr/lib/arm-linux-gnueabi libs=-lnsl -ldl -lm -lcrypt -lutil -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a gnulibc_version='2.19' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_PERLIO USE_PERL_ATOF Built under linux Compiled at Oct 15 2014 17:29:05 @INC: /var/lib/bankwatch/perl-5.20.1/lib/site_perl/5.20.1/armv5tel-linux /var/lib/bankwatch/perl-5.20.1/lib/site_perl/5.20.1 /var/lib/bankwatch/perl-5.20.1/lib/5.20.1/armv5tel-linux /var/lib/bankwatch/perl-5.20.1/lib/5.20.1 .

Wed Oct 29 20:22:58 2014 lxp [...] cpan.org - Correspondence added

This issue is causing big problems for me, as I can't find an easy workaround. Is there anything at all I can do to help progress the resolution of this issue?

Sun Nov 02 01:03:02 2014 MSCHILLI [...] cpan.org - Correspondence added

On Wed Oct 29 20:22:58 2014, LXP wrote: Show quoted text

> Is there anything at all I can do to help progress the > resolution of this issue?

6.08 works fine for me with the bank site you've reported. Is that bank the only example you have or are there others? Also, can you install Devel::Modlist and run perl -S -MDevel::Modlist lwp-request https://ibs.bankwest.com.au/ to see which module versions are being used (especially Net::Crypt, IO::Socket::SSL, or Net::SSLeay)?

Sun Nov 02 01:03:03 2014 The RT System itself - Status changed from 'new' to 'open'

Sun Nov 02 01:06:18 2014 MSCHILLI [...] cpan.org - Correspondence added

Also, one other thing to watch out of is the user agent setting, make sure to set it to something innocuous like "Mozilla/5.0 (X11; Linux x86_64)".

Sun Nov 02 03:35:07 2014 lxp [...] cpan.org - Correspondence added

Thanks for taking a look. The lwp-request command line I provided in my first message now works for me, and definitely nothing has changed on the machine running lwp- request, so something must have changed in the way the web server is serving the site. Given that wget was able to access the site at the time that lwp-request couldn't, I'd say there's still an issue with lwp-request but it's not possible to investigate further at this time. On that basis, I'm going to mark this ticket as "rejected" pending further details. In any case, the requested Devel::Modlist output follows: # perl-5.20.1/bin/perl -S -MDevel::Modlist perl-5.20.1/bin/lwp-request https://ibs.bankwest.com.au/ {{ HTML from ibs.bankwest.com.au redacted for brevity }} Use of uninitialized value in split at perl-5.20.1/lib/site_perl/5.20.1/Devel/Modlist.pm line 49. AutoLoader 5.74 Carp 1.3301 Compress::Raw::Zlib 2.065 Config 5.020001 Cwd 3.48 Encode 2.60 Encode::Alias 2.18 Encode::Config 2.05 Encode::Encoding 2.07 Encode::Locale 1.03 Errno 1.2003 Exporter 5.71 Exporter::Heavy 5.71 Fcntl 1.11 File::Basename 2.85 File::Glob 1.23 File::GlobMapper 1.000 File::Spec 3.48 File::Spec::Unix 3.48 Getopt::Long 2.42 HTML::Entities 3.69 HTML::HeadParser 3.71 HTML::Parser 3.71 HTTP::Config 6.00 HTTP::Date 6.02 HTTP::Headers 6.05 HTTP::Message 6.06 HTTP::Request 6.00 HTTP::Response 6.04 HTTP::Status 6.03 I18N::Langinfo 0.11 IO 1.31 IO::Compress::Base::Common 2.064 IO::Compress::Gzip::Constants 2.064 IO::Compress::Zlib::Extra 2.064 IO::File 1.16 IO::Handle 1.35 IO::Seekable 1.1 IO::Socket 1.37 IO::Socket::INET 1.35 IO::Socket::IP 0.29 IO::Socket::SSL 1.999 IO::Socket::SSL::PublicSuffix IO::Socket::UNIX 1.26 IO::Uncompress::Adapter::Inflate 2.064 IO::Uncompress::Base 2.064 IO::Uncompress::Gunzip 2.064 IO::Uncompress::RawInflate 2.064 LWP 6.08 LWP::MemberMixin LWP::Protocol 6.06 LWP::Protocol::http LWP::Protocol::https 6.06 LWP::UserAgent 6.06 List::Util 1.38 Mozilla::CA 20130114 Net::HTTP 6.07 Net::HTTP::Methods 6.07 Net::HTTPS 6.04 Net::SSLeay 1.66 POSIX 1.38_03 Scalar::Util 1.38 SelectSaver 1.02 Socket 2.013 Storable 2.49 Symbol 1.07 Tie::Hash 1.05 Time::Local 1.2300 URI 1.64 URI::Escape 3.31 URI::Heuristic 4.20 URI::_generic URI::_idna URI::_punycode 1.64 URI::_query URI::_server URI::http URI::https XSLoader 0.17 base 2.22 bytes 1.04 constant 1.31 integer 1.01 overload 1.22 overloading 0.02 parent 0.228 utf8 1.13_01 vars 1.03 warnings 1.23 warnings::register 1.03

Sun Nov 02 03:35:08 2014 lxp [...] cpan.org - Status changed from 'open' to 'rejected'

Sun Nov 02 03:35:08 2014 lxp [...] cpan.org - Fixed in 6.08 added

Sun Nov 02 12:13:01 2014 MSCHILLI [...] cpan.org - Correspondence added

On Sun Nov 02 03:35:07 2014, LXP wrote: Show quoted text

> Given that wget was able to access the site at the time that lwp- > request > couldn't, I'd say there's still an issue with lwp-request

Either that, or the site was intercepting lwp-request's user-agent setting and acting differently because of it. Next time you see this happen, set the user-agent explicitly to rule this case out.

Sun Nov 02 12:22:14 2014 ether [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #99523] Can't connect to ibs.bankwest.com.au:443
Date:	Sun, 2 Nov 2014 09:21:52 -0800
To:	Michael_Schilli via RT <bug-libwww-perl [...] rt.cpan.org>
From:	Karen Etheridge <ether [...] cpan.org>

On Sun, Nov 02, 2014 at 12:13:01PM -0500, Michael_Schilli via RT wrote: Show quoted text

> On Sun Nov 02 03:35:07 2014, LXP wrote:

> > Given that wget was able to access the site at the time that lwp- > > request > > couldn't, I'd say there's still an issue with lwp-request

> > Either that, or the site was intercepting lwp-request's user-agent setting and acting differently because of it. Next time you see this happen, set the user-agent explicitly to rule this case out.

When things like this happen, it's important to get a dump of the actual data sent over the wire; otherwise, it's really difficult to even begin investigating.

Sun Nov 02 18:32:18 2014 lxp [...] cpan.org - Correspondence added

On Mon Nov 03 04:13:01 2014, MSCHILLI wrote: Show quoted text

> On Sun Nov 02 03:35:07 2014, LXP wrote:

> > Given that wget was able to access the site at the time that lwp- > > request couldn't, I'd say there's still an issue with lwp-request

> > Either that, or the site was intercepting lwp-request's user-agent > setting and acting differently because of it. Next time you see this > happen, set the user-agent explicitly to rule this case out.

I forgot to mention that up until about 24 hours before I created this ticket, I was using the same libwww-perl version (or a very recent one at least) on another architecture without any problems. Nonetheless, I recognise that the server could have been configured in the interim to start filtering on User-Agent values (albeit temporarily since the server now doesn't seem to care), so I'll definitely fiddle around with that if this happens again. On Mon Nov 03 04:22:14 2014, ETHER wrote: Show quoted text

> When things like this happen, it's important to get a dump of the > actual data sent over the wire; otherwise, it's really difficult to > even begin investigating.

Karen, would you be willing to provide some basic details on how this should be done? A quick read over the libwww-perl docs suggests to me that this sort of thing would need to be done independently of libwww- perl.

Sun Nov 02 20:54:57 2014 ether [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #99523] Can't connect to ibs.bankwest.com.au:443
Date:	Sun, 2 Nov 2014 17:54:36 -0800
To:	Alex Peters via RT <bug-libwww-perl [...] rt.cpan.org>
From:	Karen Etheridge <ether [...] cpan.org>

On Sun, Nov 02, 2014 at 06:32:19PM -0500, Alex Peters via RT wrote: Show quoted text

> > When things like this happen, it's important to get a dump of the > > actual data sent over the wire; otherwise, it's really difficult to > > even begin investigating.

> > Karen, would you be willing to provide some basic details on how this > should be done? A quick read over the libwww-perl docs suggests to me > that this sort of thing would need to be done independently of libwww- > perl.

I like netstat, but there are a variety of tools depending on your architecture, operating system and vendor.