Skip Menu |

This queue is for tickets about the libwww-perl CPAN distribution.

Report information
The Basics
Id: 66838
Status: resolved
Priority: 0/
Queue: libwww-perl

People
Owner: Nobody in particular
Requestors: mschwern [...] cpan.org
Cc: nine [...] detonation.org
AdminCc:

Bug Information
Severity: Important
Broken in: 6.00
Fixed in: (no value)



Subject: Ease the https upgrade from LWP 5 to 6
I upgraded all my CPAN modules recently and started getting a mysterious "Can't verify SSL peers without knowning which Certificate Authorities to trust" error from Metabase::Client::Simple. https://rt.cpan.org/Ticket/Display.html?id=66835&results=9e0fd349dbb5faeefa02f44fb1ad332f I've tracked it down to that it appears LWP now seems to require Mozilla::CA to do https. Somehow the explanatory text got swallowed and I just saw the first line. I'm concerned that if you have a working LWP 5 setup with all the necessary modules for https, but not Mozilla::CA, that an upgrade to LWP 6 will break https. The error message is informative, but it still means an upgrade breaks stuff which can cause an outage. Is there any way to ease that transition? Can the Makefile.PL do some heuristics to see if A) you have LWP 5 installed and B) https works, maybe even just check for Crypt::SSLeay? If both are true, maybe it automatically installs the new https support modules.
I'm not sure what to do about it. One option is to not state the SSL dependencies as recommended modules, but as PREREQ_PM directly.
On Thu Mar 24 06:34:04 2011, GAAS wrote: Show quoted text
> I'm not sure what to do about it. One option is to not state the SSL > dependencies as > recommended modules, but as PREREQ_PM directly.
At a minimum, you need to update README.SSL to mention Mozilla::CA. Schwern's suggestion is (I think) to add SSL dependencies to PREREQ_PM only if you detect they are already installed. That would ensure that upgrading LWP (on a machine that already has SSL support) doesn't "break", without burdening a new LWP user with the full SSL dependency chain. -- David
I don't like to make the dependencies be dynamic based on stuff available on the system where the Makefile.PL runs. This might not be the same system where the module ends up installed. This is for instance the case for ppm in ActivePerl. Mozilla::CA can be added as a PREREQ_PM dependency for libwww-perl without much pain. That module will always build and it does not have any other dependencies by itself. That way it should always be available when/if IO::Socket::SSL is installed. Problem then is systems where Crypt::SSLeay is installed but not IO::Socket::SSL. These will start failing unless proper overrides are provided. Crypt::SSLeay can use the certificates from Mozilla::CA but it can't do full verification of hostnames.
Subject: Re: [rt.cpan.org #66838] Ease the https upgrade from LWP 5 to 6
Date: Fri, 25 Mar 2011 12:18:41 +1100
To: bug-libwww-perl [...] rt.cpan.org
From: Michael G Schwern <schwern [...] pobox.com>
Here's a solution to solve not the 5 -> 6 transition, but eliminate the problem going forward: make the https dependency explicit. That means moving LWP::Protocol::https into its own distribution that depends on LWP. LWP::Protocol::https depends on whatever encryption stuff it needs. Then if one *explicitly* wants LWP with https support they can depend on LWP and LWP::Protocol::https. No more guessing which encryption modules to depend on. On 2011.3.25 6:54 AM, Gisle_Aas via RT wrote: Show quoted text
> I'm not sure what to do about it. One option is to not state the SSL dependencies as > recommended modules, but as PREREQ_PM directly.
That would be the best solution, I'm all for encryption being as easy as possible, but I understand if you're hesitant to do that because of the track record of the encryption modules. The problem is LWP is so critical that it has to work and the encryption modules aren't quite up to snuff. I see you own the whole https chain now, so maybe one solution is to firm up the encryption modules and go with it. I think this is the best solution, but you might not want to carry that load. Another option is to split out LWP::Protocol::HTTPS, as above, and the rest of the code goes into LWP-HTTP or something. Then LWP depends on LWP-HTTP and LWP::Protocol::https. So when one installs LWP they get http and https support, but if necessary one can install just LWP-HTTP for just HTTP support. The problem there is if someone requires LWP::UserAgent they might expect HTTPS support as well. I don't have a solution for that. Show quoted text
> I don't like to make the dependencies be dynamic based on stuff available on the system where > the Makefile.PL runs. This might not be the same system where the module ends up installed. > This is for instance the case for ppm in ActivePerl.
I understand that concern. IMO vendors have the infrastructure and scale to deal with special cases like this. Provide a --with-ssl flag to the Makefile.PL. They can set explicitly and then they'll add that to their build system. Individual users are not set up to handle special cases. Each one will expect that just installing from the CPAN shell will work. For example, MakeMaker ships with a bunch of bundled modules in inc/. Vendors have to know to delete inc, or have those bundled modules already installed, before packaging MakeMaker. I'm ok with that because inc lets CPAN shell installs work. In short, I'm happy to make life a bit more difficult for vendors if it makes life simpler for individual CPAN users. Show quoted text
> Mozilla::CA can be added as a PREREQ_PM dependency for libwww-perl without much pain. > That module will always build and it does not have any other dependencies by itself. That way it > should always be available when/if IO::Socket::SSL is installed.
That sounds like an ok solution if it works. Show quoted text
> Problem then is systems where Crypt::SSLeay is installed but not IO::Socket::SSL. These > will start failing unless proper overrides are provided. Crypt::SSLeay can
use the Show quoted text
> certificates from Mozilla::CA but it can't do full verification of hostnames.
So if I get that straight, if the user has just Crypt::SSLeay and Mozilla::CA there's additional fiddling with LWP flags that has to go on before it will work? Could those flags be the default for Crypt::SSLeay? Is there enough benefit supporting two SSL chains in the code? -- 91. I am not authorized to initiate Jihad. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/
I like the solution of splitting out LWP::Protocol::https into it's own separate distribution with hard dependencies on the required SSL modules best. It try that.
From: ludwig.nussel [...] suse.de
What's the reason to use Mozilla::CA anways? Linux distributions have a set of system certificates in /etc/ssl/certs so no need use your own CA bundle.
On Mon Mar 28 09:05:11 2011, ludwig.nussel@suse.de wrote: Show quoted text
> What's the reason to use Mozilla::CA anways? Linux distributions have a > set of system certificates in /etc/ssl/certs so no need use your own CA > bundle.
This may come as a shock to you, but people use things other than Linux or even Unix! Even if they use Unix, are SSL certs installed? Are they up to date? Are they in a format we understand? Is /etc/ssl/certs where they put it? It might be possible to look around and see if there's something useful on the filesystem if Mozilla::CA isn't there. That would probably make vendor packaging easier.
On Mon Mar 28 09:05:11 2011, ludwig.nussel@suse.de wrote: Show quoted text
> What's the reason to use Mozilla::CA anways? Linux distributions have a > set of system certificates in /etc/ssl/certs so no need use your own CA > bundle.
The plan is in fact to introduce some module that use heuristics to determine if we can use OS-provided certs and then only fall back on Mozilla::CA when that fails. That module has not been written yet, but I want to plug it in eventually. The plan was to start out with something that we know will work regardless of platform and then iterate from there. ... and even if we find some cert files under /etc/ssl/certs it might not be the best source if they are less up-to-date than the certs of Mozilla::CA. BTW, on my Linux system I find the OS provided certs under /etc/pki/tls/cert.pem.
I've now created the LWP-Protocol-https distribution and unbundled the file from libwww-perl- 6.02. Let's see to what degree this confuses people.
From: ludwig.nussel [...] suse.de
Am Mo 28. Mär 2011, 15:47:56, GAAS schrieb: Show quoted text
> On Mon Mar 28 09:05:11 2011, ludwig.nussel@suse.de wrote:
> > What's the reason to use Mozilla::CA anways? Linux distributions
> have a
> > set of system certificates in /etc/ssl/certs so no need use your own
> CA
> > bundle.
> > The plan is in fact to introduce some module that use heuristics to
determine Show quoted text
> if we can use OS-provided certs and then only fall back on Mozilla::CA
when Show quoted text
> that fails. That module has not been written yet, but I want to plug
it in Show quoted text
> eventually. The plan was to start out with something that we know
will work Show quoted text
> regardless of platform and then iterate from there.
The best fix would be to export SSL_CTX_set_default_verify_paths() which just uses the openssl compiled in defaults in Net::SSLeay IMO. IO::Socket::SSL could call that already so higher level module don't need to implement workarounds.
On Mon Mar 28 10:14:04 2011, MSCHWERN wrote: Show quoted text
> Even if they use Unix, are SSL certs installed? Are they up to date? > Are they in a format we understand? Is /etc/ssl/certs where they put it?
When someone is concerned enough about security to use SSL, chances are, that this person also installs security updates for his system, so those certificates ought to be up to date. More up to date than some random Perl module that gets installed as dependency once and never touched again till Perl itself gets updated to a new version and the installing of dependencies starts again. And in contrast to some obscure file somewhere in /usr/lib/perl5/... there are documented ways to add CAs to /etc/ssl. Ways that are used in practice. So a simple install of dependencies of a Perl module led to updating LWP breaking a lot of Perl based tools that suddenly don't want to talk to our servers anymore because they cannot verify certificates despite CA certificates being installed in the correct location for the system. Great.
Subject: Re: [rt.cpan.org #66838] Ease the https upgrade from LWP 5 to 6
Date: Tue, 10 May 2011 10:51:43 +1000
To: bug-libwww-perl [...] rt.cpan.org
From: Michael G Schwern <schwern [...] pobox.com>
On 2011.5.9 10:05 PM, Stefan Seifert via RT wrote: Show quoted text
> On Mon Mar 28 10:14:04 2011, MSCHWERN wrote:
>> Even if they use Unix, are SSL certs installed? Are they up to date? >> Are they in a format we understand? Is /etc/ssl/certs where they put it?
> > When someone is concerned enough about security to use SSL, chances > are, that this person also installs security updates for his system, so > those certificates ought to be up to date. More up to date than some > random Perl module that gets installed as dependency once and never > touched again till Perl itself gets updated to a new version and the > installing of dependencies starts again.
This may be true of someone running an HTTPS *server* [1], but certainly not to a client simply linking to an https URL. That's *minimal* security, often simply used because its the URL they were given, and probably not running on the same machine it was written on. For example, a Twitter client. No, I'll wager your average user of http links has no idea what the state of the security of their system is. Even then, the up-to-date-ness of the files is only one consideration, and not even the major one. Do we know where they're installed? Are they in a format we can read? Are the ones we found the ones that are kept up to date or are they some old duplicates we happened to find grovelling around the filesystem? This is essentially the same argument as to whether DateTime should ship its own time zone files or use the system ones. I think the integration of certs is best left to the vendors. If an OS vendor wants to ship a copy of the module that links to their own cert system, ok. I think that's how Debian resolved it. The CPAN module doesn't now what environment it lives in and should be independent. Now, it *could* make an effort to try out likely locations of system certs, perhaps ignore them if they're out of date, but I don't know if that's possible. That would be a place to put effort. [1] Even then I'd argue it isn't, but less strongly. -- 52. Not allowed to yell "Take that Cobra" at the rifle range. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/