Bug #102640 for Net-SSLeay: HTTP POST requests over 90kb fail with OpenSSL 1.0.2

Sun Mar 08 11:45:59 2015 KENTNL [...] cpan.org - Ticket created

Subject:

HTTP POST requests over 90kb fail with OpenSSL 1.0.2

I've had a hard time nailing this issue down as to where its actually going wrong.

All the facts I know are, that this issue was not present on 1.0.1l, and is present on 1.0.2

Under 1.0.2 , https://metacpan.org/source/ETHER/libwww-perl-6.13/lib/LWP/Protocol/http.pm#L282 <-- this line fails when the socket is an IO::Socket::SSL socket.

Using an MITM to unwrap the SSL and provide plain HTTP to the same requests results in no error.

Downgrading openssl to 1.0.1l also results in no error.

The error dies with $! being "write failed: Bad address" which is a C Level error to do with bad pointers ( usually ).

And I've found similar issues with similar upload thresholds triggering the same Bad address error on ruby projects ( https://github.com/excon/excon/issues/467 )

But there's no obvious smoking gun, there's no related articles I can see at OpenSSL's official forums, so at best I might assume there's some kind of ABI missmatch 1.0.2 introduces that needs to be adjusted for.

Unfortunately I don't know what other data might be helpful in replicating this issue, other than to say that uploading the following files to PAUSE using cpan-uploader ( LWP -> IO::Socket::SSL -> Net::SSLeay ) reliably replicated the failure, and circumvention was required to get uploads working:

https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Color-Library-Dictionary-XKCD-1.000000.tar.gz

https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Dist-Zilla-PluginBundle-Author-KENTNL-2.024001.tar.gz
https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Dist-Zilla-PluginBundle-Author-KENTNL-2.024000.tar.gz

Sun Mar 08 17:32:19 2015 ether [...] cpan.org - Cc ETHER added

Mon Mar 09 01:34:18 2015 mikem [...] airspayce.com - Correspondence added

Subject:	Re: [rt.cpan.org #102640] HTTP POST requests over 90kb fail with OpenSSL 1.0.2
Date:	Mon, 09 Mar 2015 15:34:05 +1000
To:	bug-Net-SSLeay [...] rt.cpan.org
From:	Mike McCauley <mikem [...] airspayce.com>

Hello Kent, I havent been able to reproduce this with cpan-upload If you can provide some sample code or command line to demonstrate the problem I will do my best with it. Cheers. On Sunday, March 08, 2015 11:46:00 AM you wrote: Show quoted text

> Sun Mar 08 11:45:59 2015: Request 102640 was acted upon. > Transaction: Ticket created by KENTNL > Queue: Net-SSLeay > Subject: HTTP POST requests over 90kb fail with OpenSSL 1.0.2 > Broken in: 1.68 > Severity: (no value) > Owner: Nobody > Requestors: KENTNL@cpan.org > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102640 > > > > I've had a hard time nailing this issue down as to where its actually going > wrong. > > All the facts I know are, that this issue was not present on 1.0.1l, and is > present on 1.0.2 > > Under 1.0.2 , > https://metacpan.org/source/ETHER/libwww-perl-6.13/lib/LWP/Protocol/http.pm# > L282 <-- this line fails when the socket is an IO::Socket::SSL socket. > > Using an MITM to unwrap the SSL and provide plain HTTP to the same requests > results in no error. > > Downgrading openssl to 1.0.1l also results in no error. > > The error dies with $! being "write failed: Bad address" which is a C Level > error to do with bad pointers ( usually ). > > And I've found similar issues with similar upload thresholds triggering the > same Bad address error on ruby projects ( > https://github.com/excon/excon/issues/467 ) > > But there's no obvious smoking gun, there's no related articles I can see at > OpenSSL's official forums, so at best I might assume there's some kind of > ABI missmatch 1.0.2 introduces that needs to be adjusted for. > > Unfortunately I don't know what other data might be helpful in replicating > this issue, other than to say that uploading the following files to PAUSE > using cpan-uploader ( LWP -> IO::Socket::SSL -> Net::SSLeay ) reliably > replicated the failure, and circumvention was required to get uploads > working: > > https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Color-Library-Dictionary-XK > CD-1.000000.tar.gz > > https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Dist-Zilla-PluginBundle-Aut > hor-KENTNL-2.024001.tar.gz > https://cpan.metacpan.org/authors/id/K/KE/KENTNL/Dist-Zilla-PluginBundle-Au > thor-KENTNL-2.024000.tar.gz

-- Mike McCauley VK4AMM mikem@airspayce.com Airspayce Pty Ltd 9 Bulbul Place Currumbin Waters QLD 4223 Australia http://www.airspayce.com Phone +61 7 5598-7474

Mon Mar 09 01:34:20 2015 The RT System itself - Status changed from 'new' to 'open'

Wed Mar 11 21:40:28 2015 KENTNL [...] cpan.org - Correspondence added

Earlier today RJBS was seen on #toolchain having the same issue. But it appeared semi-transient for him.

I've been trying to make a reduced case, but the code I'm trying to simplify and retain the failure condition is way way beyond my mental capacity to process and I've not been able to get anyone to assist me.

I need to find a way to reduce the code to its absolute minimum retaining the PAUSE failure, and then when I'm sure the code can be simplified no more, I have to make a non-pause server that assists in replicating the failure.

And then I have to work out what series of OpenSSL compilation options I happen to have that built my system OpenSSL install that cause the issue.

But I haven't even done 1/3 :(

Mon May 25 14:17:16 2015 Steffen_Ullrich [...] genua.de - Correspondence added

I have the feeling that this problem might be related to CVE-2015-0290 "OpenSSL Multiblock Corrupted Pointer Issue". The diff at https://git.openssl.org/?p=openssl.git;a=commitdiff;h=77c77f0a1b9f15b869ca3342186dfbedd1119d0e;hp=8b84495380098592ef7bb2fa9209ccb87803bf1d indicates for me that this situation might happen sometimes with a combination of a multiblock-capable cipher (AES-NI) and non-blocking I/O when the server does not read data fast enough so that the socket buffer will fill up and BIO_write will not be able to write the full SSL fragment to the socket. This would explain also if it is happening with larger data only, because it takes some amount of data to fill up the socket buffer. This OpenSSL problem was fixed in 1.02a which was only released after this bug report. So it would be good to know if this problem still happens with OpenSSL 1.02a.

Mon May 25 14:28:47 2015 KENTNL [...] cpan.org - Correspondence added

>
> This OpenSSL problem was fixed in 1.02a which was only released after
> this bug report. So it would be good to know if this problem still
> happens with OpenSSL 1.02a.

Indeed. I haven't personally seen it myself in a long time now, ( having upgraded to 1.0.2a around March 21 ) which is good because it happened repeatedly before.

The ruby instance I referenced with the same error also appears to be solved in the same way.

So I'm relatively comfortable in putting this one to bed and just having some kind of "Your OpenSSL library version is in a list of ones known to be critically broken, plz upgrade" advice.

Tue May 26 01:51:15 2015 MIKEM [...] cpan.org - Correspondence added

Just to reiterate: apparently due to a bug in OPenSSL that is fixed in OpenSSL 1.0.2a. Thanks to Steffen Ulrich for shedding light on this.

Tue May 26 01:51:24 2015 MIKEM [...] cpan.org - Status changed from 'open' to 'rejected'

Tue May 26 01:51:24 2015 MIKEM [...] cpan.org - Taken

Tue May 26 02:17:24 2015 SREZIC [...] cpan.org - Correspondence added

On 2015-05-26 01:51:15, MIKEM wrote: Show quoted text

> Just to reiterate: > apparently due to a bug in OPenSSL that is fixed in OpenSSL 1.0.2a.

Maybe this issue could be mentioned in Net::SSLeay's Pod page, in the "KNOWN BUGS AND CAVEATS" section?

Tue May 26 02:22:59 2015 mikem [...] airspayce.com - Correspondence added

Subject:	Re: [rt.cpan.org #102640] HTTP POST requests over 90kb fail with OpenSSL 1.0.2
Date:	Tue, 26 May 2015 16:22:49 +1000
To:	bug-Net-SSLeay [...] rt.cpan.org
From:	Mike McCauley <mikem [...] airspayce.com>

Done. Cheers. On Tuesday, May 26, 2015 02:17:25 AM you wrote: Show quoted text

> Queue: Net-SSLeay > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=102640 > > > On 2015-05-26 01:51:15, MIKEM wrote:

> > Just to reiterate: > > apparently due to a bug in OPenSSL that is fixed in OpenSSL 1.0.2a.

> > Maybe this issue could be mentioned in Net::SSLeay's Pod page, in the "KNOWN > BUGS AND CAVEATS" section?

-- Mike McCauley VK4AMM mikem@airspayce.com Airspayce Pty Ltd 9 Bulbul Place Currumbin Waters QLD 4223 Australia http://www.airspayce.com Phone +61 7 5598-7474