Bug #87163 for libwww-perl: perl-5.18.0 + LWP-6.05 fails to send User-Agent Header

Mon Jul 22 13:07:55 2013 david [...] davidfavor.com - Ticket created

Subject:	perl-5.18.0 + LWP-6.05 fails to send User-Agent Header
Date:	Mon, 22 Jul 2013 12:07:37 -0500
To:	" bug-libwww-perl" [...] rt.cpan.org
From:	David Favor <david [...] davidfavor.com>

Very ugly. Output from my test code. ./lwp-bug using: perl-5.18.0 + LWP-6.05 agent: expected : Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36 agent: from LWP : libwww-perl/6.05 agent: live check: libwww-perl/6.05 This code checks ->_agent + does a live check against http://WhatsMyUserAgent.com and returns results. Tried with several different perls. Code follows. Show quoted text

_______ #!/usr/bin/env perl # author : David Favor @ http://DavidFavor.com # purpose: Illustrate LWP fails to send User-Agent Header use strict; use warnings; use Carp qw/confess/; use Config; use LWP::UserAgent; use HTTP::Request::Common qw(GET POST); use Data::Dumper; $Data::Dumper::Useqq = 1; $Data::Dumper::Terse = 1; $Data::Dumper::Sortkeys = 1; my $bua = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36'; my $perlver = $Config{api_versionstring}; my $lwpver = $LWP::VERSION; print "using: perl-$perlver + LWP-$lwpver\n"; my $lwp = new LWP::UserAgent; $lwp->agent ($bua); my $url = 'http://WhatsMyUserAgent.com/'; my $req = GET $url; my $res = $lwp->request($req); confess("Unable to GET($url): ", $res->status_line) unless $res->is_success; my $sua = $res->content; $sua =~ s/^.+<span id="body_lbUserAgent">//o; $sua =~ s/<span.+$//o; print "agent: expected : ", $bua, "\n"; print "agent: from LWP : ", $lwp->_agent, "\n"; print "agent: live check: ", $lwp->_agent, "\n"; -- Love Living Well Doing What You Love? http://DavidFavor.com/bb can help!

Fri Jul 26 09:36:26 2013 david [...] davidfavor.com - Correspondence added

Subject:	Re: [rt.cpan.org #87163] AutoReply: perl-5.18.0 + LWP-6.05 fails to send User-Agent Header
Date:	Fri, 26 Jul 2013 08:36:10 -0500
To:	bug-libwww-perl [...] rt.cpan.org
From:	David Favor <david [...] davidfavor.com>

This problems stretches back to LWP-5.x too, so maybe my code is incorrect. Nothing I do seems to *actually* effect the User Agent. Changes effect LWP data structures and the actual, sent User-Agent header never changes. -- Love Living Well Doing What You Love? http://DavidFavor.com/bb can help!

Fri Jul 26 10:42:50 2013 ether [...] cpan.org - Correspondence added

Subject:	Re: [rt.cpan.org #87163] AutoReply: perl-5.18.0 + LWP-6.05 fails to send User-Agent Header
Date:	Fri, 26 Jul 2013 07:42:34 -0700
To:	David Favor via RT <bug-libwww-perl [...] rt.cpan.org>
From:	Karen Etheridge <ether [...] cpan.org>

On Fri, Jul 26, 2013 at 09:36:26AM -0400, David Favor via RT wrote: Show quoted text

> This problems stretches back to LWP-5.x too, so maybe my code is incorrect. > Nothing I do seems to *actually* effect the User Agent. > Changes effect LWP data structures and the actual, sent User-Agent header never changes.

Can you boil this down into a simple reproducable example?

Fri Jul 26 10:42:51 2013 The RT System itself - Status changed from 'new' to 'open'

Mon Jul 29 08:45:09 2013 david [...] davidfavor.com - Correspondence added

Subject:	Re: [rt.cpan.org #87163] AutoReply: perl-5.18.0 + LWP-6.05 fails to send User-Agent Header
Date:	Mon, 29 Jul 2013 07:44:53 -0500
To:	bug-libwww-perl [...] rt.cpan.org
From:	David Favor <david [...] davidfavor.com>

Karen Etheridge via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=87163 > > > On Fri, Jul 26, 2013 at 09:36:26AM -0400, David Favor via RT wrote:

>> This problems stretches back to LWP-5.x too, so maybe my code is incorrect. >> Nothing I do seems to *actually* effect the User Agent. >> Changes effect LWP data structures and the actual, sent User-Agent header never changes.

> > Can you boil this down into a simple reproducable example? > >

Apologies. I should have attached the code. Here it is... Show quoted text

_______ #!/usr/bin/env perl # author : David Favor @ http://DavidFavor.com # purpose: Illustrate LWP fails to send User-Agent Header use strict; use warnings; use Carp qw/confess/; use Config; use LWP::Debug qw(+); use LWP::UserAgent; use HTTP::Request::Common qw(GET POST); use Data::Dumper; $Data::Dumper::Useqq = 1; $Data::Dumper::Terse = 1; $Data::Dumper::Sortkeys = 1; my $bua = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36'; my $perlver = $Config{api_versionstring}; my $lwpver = $LWP::VERSION; print "using: perl-$perlver + LWP-$lwpver\n"; my $lwp = new LWP::UserAgent; $lwp->agent ($bua); my $url = 'http://WhatsMyUserAgent.com/'; my $req = GET $url; my $res = $lwp->request($req); confess("Unable to GET($url): ", $res->status_line) unless $res->is_success; my $sua = $res->content; $sua =~ s/^.+<span id="body_lbUserAgent">//o; $sua =~ s/<span.+$//o; print "agent: expected : ", $bua, "\n"; print "agent: from LWP : ", $lwp->_agent, "\n"; print "agent: live check: ", $lwp->_agent, "\n"; -- Love Living Well Doing What You Love? http://DavidFavor.com/bb can help!

Sat Oct 05 08:01:42 2013 david [...] davidfavor.com - Correspondence added

Subject:	Re: [rt.cpan.org #87163] AutoReply: perl-5.18.0 + LWP-6.05 fails to send User-Agent Header
Date:	Sat, 05 Oct 2013 07:01:18 -0500
To:	bug-libwww-perl [...] rt.cpan.org
From:	David Favor <david [...] davidfavor.com>

The problem appears to be in the class LWP::UserAgent.pm in the method _agent() which returns "libwww-perl/$LWP::VERSION"... so the hack to temporarily fix this problem is to hack this .pm file and change the UA to a different value. ack -r -l _agent in the LWP directory shows this instance variable is never changed anywhere, hence the UA never changes. Maybe I'm missing something... as it stands, it appears there is no possible way to change UA value. Refer to my previous message for a short script that sets UA according to docs, then checks value against what is actually seen in a browser. This code bug also explains why setting UA in all sorts of tools gets all these tools blocked. Most tools change UA to a recent browser, to circumvent sites that block libwww UA values. -- Love Living Well Doing What You Love? http://DavidFavor.com/books can help!

Sat Oct 05 09:44:58 2013 $_ = 'spro^^*%*^6ut# [...] &$%*c>#!^!#&!pan.org'; y/a-z.@//cd; print - Correspondence added

On Mon Jul 22 13:07:55 2013, david@davidfavor.com wrote: Show quoted text

> Very ugly. > > Output from my test code. > > ./lwp-bug > using: perl-5.18.0 + LWP-6.05 > agent: expected : Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 > Safari/537.36 > agent: from LWP : libwww-perl/6.05 > agent: live check: libwww-perl/6.05 > > This code checks ->_agent + does a live check against > http://WhatsMyUserAgent.com and returns results. > > Tried with several different perls.

I’ve been doing this for years and have never had the problem you describe. Show quoted text

> > Code follows. > _______ > > #!/usr/bin/env perl > > # author : David Favor @ http://DavidFavor.com > # purpose: Illustrate LWP fails to send User-Agent Header > > use strict; > use warnings; > > use Carp qw/confess/; > > use Config; > use LWP::UserAgent; > use HTTP::Request::Common qw(GET POST); > > use Data::Dumper; > $Data::Dumper::Useqq = 1; > $Data::Dumper::Terse = 1; > $Data::Dumper::Sortkeys = 1; > > my $bua = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 > Safari/537.36'; > > my $perlver = $Config{api_versionstring}; > my $lwpver = $LWP::VERSION; > > print "using: perl-$perlver + LWP-$lwpver\n"; > > my $lwp = new LWP::UserAgent; > $lwp->agent ($bua); > > my $url = 'http://WhatsMyUserAgent.com/'; > my $req = GET $url; > my $res = $lwp->request($req); > confess("Unable to GET($url): ", $res->status_line) unless $res-

> >is_success;

> > my $sua = $res->content; > $sua =~ s/^.+<span id="body_lbUserAgent">//o; > $sua =~ s/<span.+$//o; > > print "agent: expected : ", $bua, "\n"; > print "agent: from LWP : ", $lwp->_agent, "\n";

->_agent (as opposed to ->agent) is documented to return the default user agent string. Show quoted text

> print "agent: live check: ", $lwp->_agent, "\n";

Try printing $sua here.

Sun Oct 06 07:55:12 2013 GAAS [...] cpan.org - Correspondence added

I ran the test script and it behaves as expected. The $sua value does contain the expected string. Especially if you tweaks the regexp so that they actually clean up the content.

Sun Oct 06 07:55:12 2013 GAAS [...] cpan.org - Status changed from 'open' to 'rejected'

Sat Oct 19 08:03:27 2013 david [...] davidfavor.com - Correspondence added

Subject:	Re: [rt.cpan.org #87163] perl-5.18.0 + LWP-6.05 fails to send User-Agent Header
Date:	Sat, 19 Oct 2013 07:03:05 -0500
To:	bug-libwww-perl [...] rt.cpan.org
From:	David Favor <david [...] davidfavor.com>

Father Chrysostomos via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=87163 > > > On Mon Jul 22 13:07:55 2013, david@davidfavor.com wrote:

>> Very ugly. >> >> Output from my test code. >> >> ./lwp-bug >> using: perl-5.18.0 + LWP-6.05 >> agent: expected : Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 >> Safari/537.36 >> agent: from LWP : libwww-perl/6.05 >> agent: live check: libwww-perl/6.05 >> >> This code checks ->_agent + does a live check against >> http://WhatsMyUserAgent.com and returns results. >> >> Tried with several different perls.

> > I’ve been doing this for years and have never had the problem you describe. >

>> Code follows. >> _______ >> >> #!/usr/bin/env perl >> >> # author : David Favor @ http://DavidFavor.com >> # purpose: Illustrate LWP fails to send User-Agent Header >> >> use strict; >> use warnings; >> >> use Carp qw/confess/; >> >> use Config; >> use LWP::UserAgent; >> use HTTP::Request::Common qw(GET POST); >> >> use Data::Dumper; >> $Data::Dumper::Useqq = 1; >> $Data::Dumper::Terse = 1; >> $Data::Dumper::Sortkeys = 1; >> >> my $bua = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 >> Safari/537.36'; >> >> my $perlver = $Config{api_versionstring}; >> my $lwpver = $LWP::VERSION; >> >> print "using: perl-$perlver + LWP-$lwpver\n"; >> >> my $lwp = new LWP::UserAgent; >> $lwp->agent ($bua); >> >> my $url = 'http://WhatsMyUserAgent.com/'; >> my $req = GET $url; >> my $res = $lwp->request($req); >> confess("Unable to GET($url): ", $res->status_line) unless $res-

>>> is_success;

>> my $sua = $res->content; >> $sua =~ s/^.+<span id="body_lbUserAgent">//o; >> $sua =~ s/<span.+$//o; >> >> print "agent: expected : ", $bua, "\n"; >> print "agent: from LWP : ", $lwp->_agent, "\n";

> > ->_agent (as opposed to ->agent) is documented to return the default user agent string. >

>> print "agent: live check: ", $lwp->_agent, "\n";

> > Try printing $sua here.

Ah... $lwp->agent is the trick... Thanks! -- Love Living Well Doing What You Love? http://DavidFavor.com/books can help!