Bug #90414 for HTTP-Proxy: HTTPS data transfer fails

Fri Nov 15 11:58:44 2013 vincenzo.buttazzo [...] altervista.it - Ticket created

Subject:	HTTPS data transfer fails
Date:	Fri, 15 Nov 2013 18:02:55 +0100
To:	bug-HTTP-Proxy [...] rt.cpan.org
From:	"Vincenzo Buttazzo" <vincenzo.buttazzo [...] altervista.it>

Hi, we're using HTTP::Proxy to connect PHP scripts on our shared hosting with the internet. In this context a customized version of HTTP::Proxy is used to block connections to undesired domains. Things work good except for HTTPS connections because with some web servers we don't got any data back after the protocol handshake and sending POST data. After some investigation we found out that the size of POST data was related to the problem and finally we found a solution adding this line after the opening of the $upstream socket in the _handle_CONNECT function (around row 600), just after the $upstream check block: $upstream->setsockopt(SOL_SOCKET, SO_SNDBUF, $conn->getsockopt(SOL_SOCKET, SO_RCVBUF)); In fact in our context the $upstream's "send buffer" was smaller than the packets incoming from $conn, so data packets sent from PHP to HTTP::Proxy where splitted when relayed to the external website. Our supposition is that this behavior breaks someway the HTTP protocol, but the fix works good and now HTTPS works for all websites. The problem was not related to our customization because we had the same problem using a vanilla HTTP::Proxy. So, thanks for your great module, and I hope my hint can be useful for improving the module. Bye! -- Vincenzo Buttazzo Responsabile sviluppo web http://www.altervista.org/

Sun Apr 12 10:27:34 2015 MSTEMLE [...] cpan.org - Correspondence added

On Fri Nov 15 11:58:44 2013, vincenzo.buttazzo@altervista.it wrote: Show quoted text

> Hi, > we're using HTTP::Proxy to connect PHP scripts on our shared hosting with > the internet. In this context a customized version of HTTP::Proxy is used > to block connections to undesired domains. > > Things work good except for HTTPS connections because with some web > servers we don't got any data back after the protocol handshake and > sending POST data. > > After some investigation we found out that the size of POST data was > related to the problem and finally we found a solution adding this line > after the opening of the $upstream socket in the _handle_CONNECT function > (around row 600), just after the $upstream check block: > > $upstream->setsockopt(SOL_SOCKET, SO_SNDBUF, $conn->getsockopt(SOL_SOCKET, > SO_RCVBUF)); > > In fact in our context the $upstream's "send buffer" was smaller than the > packets incoming from $conn, so data packets sent from PHP to HTTP::Proxy > where splitted when relayed to the external website. Our supposition is > that this behavior breaks someway the HTTP protocol, but the fix works > good and now HTTPS works for all websites. > > The problem was not related to our customization because we had the same > problem using a vanilla HTTP::Proxy. > > > So, thanks for your great module, and I hope my hint can be useful for > improving the module. > > Bye! > >

I'm not sure if this is the best module to use for HTTPS proxying, but also you could be shrinking the size of the send buffer, making it truncate portions of the transmission. I would think that the data you send to the proxy would end up being larger than that which you received since you're adding proxy headers as well. The module `IO::Stream::Proxy::HTTPS` seems like it might be a better option for proxying HTTPS. Does this sound like gibberish?

Sun Apr 12 10:27:34 2015 The RT System itself - Status changed from 'new' to 'open'

Mon Apr 13 05:53:32 2015 vincenzo.buttazzo [...] altervista.it - Correspondence added

Subject:	Re: [rt.cpan.org #90414] HTTPS data transfer fails
Date:	Mon, 13 Apr 2015 11:53:17 +0200
To:	bug-HTTP-Proxy [...] rt.cpan.org
From:	Vincenzo Buttazzo <vincenzo.buttazzo [...] altervista.it>

Il 12/04/2015 16:27, Michael D. Stemle, Jr. via RT ha scritto: Show quoted text

> I'm not sure if this is the best module to use for HTTPS proxying,

Maybe, but it supports HTTP and HTTPS at the same time and that's our case. Show quoted text

> but also you could be shrinking the size of the send buffer, making it > truncate portions of the transmission.

Yes, that's the other way. The important thing is that both socket's buffers have to be of the same size. Show quoted text

> I would think that the data you send to the proxy would end up being > larger than that which you received since you're adding proxy headers > as well.

Right, but the proxy first receives the CONNECT header and then opens a socket to the destination. From that point the data sent to the proxy will be passed unchanged from the downstream socket to the upstream socket and viceversa, so the buffer sizes of this sockets have to be the same to not break the encrypted stream into unusable chunks. -- Vincenzo Buttazzo Sviluppo Altervista + GialloZafferano

Tue Apr 14 12:52:47 2015 book [...] cpan.org - Correspondence added

On Fri Nov 15 11:58:44 2013, vincenzo.buttazzo@altervista.it wrote: Show quoted text

> > Things work good except for HTTPS connections because with some web > servers we don't got any data back after the protocol handshake and > sending POST data. > > After some investigation we found out that the size of POST data was > related to the problem and finally we found a solution adding this line > after the opening of the $upstream socket in the _handle_CONNECT function > (around row 600), just after the $upstream check block: > > $upstream->setsockopt(SOL_SOCKET, SO_SNDBUF, $conn->getsockopt(SOL_SOCKET, > SO_RCVBUF)); >

I've applied this change, based on my understanding of the above paragraph (a patch would have been less ambiguous ;-): https://github.com/book/HTTP-Proxy/commit/a3c2514d779d3335d947a52a614d5649bcd31bd0 Please let me know if it works for you, so that I can cut a new release. Thanks for your feedback, -- BooK

Tue Apr 21 07:50:41 2015 vincenzo.buttazzo [...] altervista.it - Correspondence added

Subject:	Re: [rt.cpan.org #90414] HTTPS data transfer fails
Date:	Tue, 21 Apr 2015 13:50:26 +0200
To:	bug-HTTP-Proxy [...] rt.cpan.org
From:	Vincenzo Buttazzo <vincenzo.buttazzo [...] altervista.it>

Sorry for the late reply and the not so precise indication. Anyway that's exactly the same patch we have used in the last 2 years! Il 14/04/2015 18:52, Philippe Bruhat (BooK) via RT ha scritto: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=90414 > > > On Fri Nov 15 11:58:44 2013, vincenzo.buttazzo@altervista.it wrote:

>> Things work good except for HTTPS connections because with some web >> servers we don't got any data back after the protocol handshake and >> sending POST data. >> >> After some investigation we found out that the size of POST data was >> related to the problem and finally we found a solution adding this line >> after the opening of the $upstream socket in the _handle_CONNECT function >> (around row 600), just after the $upstream check block: >> >> $upstream->setsockopt(SOL_SOCKET, SO_SNDBUF, $conn->getsockopt(SOL_SOCKET, >> SO_RCVBUF)); >>

> I've applied this change, based on my understanding of the above paragraph > (a patch would have been less ambiguous ;-): > > https://github.com/book/HTTP-Proxy/commit/a3c2514d779d3335d947a52a614d5649bcd31bd0 > > Please let me know if it works for you, so that I can cut a new release. > > Thanks for your feedback, > > -- BooK >

-- Vincenzo Buttazzo Sviluppo Altervista + GialloZafferano

Tue Apr 21 16:30:26 2015 book [...] cpan.org - Correspondence added

On Tue Apr 21 07:50:41 2015, vincenzo.buttazzo@altervista.it wrote: Show quoted text

> Sorry for the late reply and the not so precise indication. > > Anyway that's exactly the same patch we have used in the last 2 years! >

Thanks! It will be in the next release, then. -- BooK

Tue Jun 09 10:46:25 2015 ppisar [...] redhat.com - Correspondence added

From:

ppisar [...] redhat.com

Dne Út 21.dub.2015 16:30:26, BOOK napsal(a): Show quoted text

> On Tue Apr 21 07:50:41 2015, vincenzo.buttazzo@altervista.it wrote:

> > Sorry for the late reply and the not so precise indication. > > > > Anyway that's exactly the same patch we have used in the last 2 years! > >

> > Thanks! It will be in the next release, then. >

This is in released 0.303 version. I think this bug report can be closed.

Mon Jun 15 20:25:44 2015 book [...] cpan.org - Status changed from 'open' to 'resolved'

Mon Jun 15 20:25:45 2015 book [...] cpan.org - Fixed in 0.303 added