Skip Menu |

This queue is for tickets about the HTTP-Proxy CPAN distribution.

Report information
The Basics
Id: 13877
Status: open
Priority: 0/
Queue: HTTP-Proxy

People
Owner: Nobody in particular
Requestors: chris+rt [...] chrisdolan.net
Cc:
AdminCc:

Bug Information
Severity: Wishlist
Broken in: 0.15
Fixed in: (no value)



Subject: Allow buffers in filter_last
I'm trying to implement HTTP::Proxy::BodyFilter::throttle and have hit a limitation of HTTP::Proxy. My filter tries to emit a fixed maximum number of bytes per second. It accomplishes this via the filter() method by deferring some content to the buffer until enough time has passed (using sleep() perhaps). However, when the proxy has received all of the upstream data, it no longer permits use of the buffer and requires the filter to finish immediately. I request that a while() loop be added around the filtering loop once the content is all received. That while loop should continue until all of the buffers are flushed. Something roughly like this would work (untested): sub filter_last { my $self = shift; return unless $self->{body}; # sanity check my $i = 0; my ( $data, $message, $protocol ) = @_; for ( @{ $self->{current} } ) { $$data = ${ $self->{buffers}[ $i ] } . $$data; ${ $self->{buffers}[ $i++ ] } = ""; $_->filter( $data, $message, $protocol, undef ); } # call the cleanup routine if needed for ( @{ $self->{current} } ) { $_->end if $_->can('end'); } # clean up the mess for next time $self->eod; }
From: cdolan [...] cpan.org
(oops hit return too soon in that last post) sub filter_last { my $self = shift; my $conn = shift; return unless $self->{body}; # sanity check my $i = 0; my ( $data, $message, $protocol ) = @_; while ($$data ne "" && grep({$_ ne ""} @{$self->{buffers}}) > 0) { $self->filter($data, $message, $protocol); if ( length $data ) { if ($chunked) { printf $conn "%x$CRLF%s$CRLF", length($data), $data; } else { print $conn $data; } } } # call the cleanup routine if needed for ( @{ $self->{current} } ) { $_->end if $_->can('end'); } # clean up the mess for next time $self->eod; }
From: cdolan [...] cpan.org
And here's a working version (attached). It rather hackish, primarily in its determination of $chunked, but functional.
sub filter_last { my $self = shift; return unless $self->{body} && $self->{current} && @{$self->{current}} > 0; my $conn = $self->{current}->[0]->proxy->client_socket; # hack... my $chunked = $conn->proto_ge("HTTP/1.1"); # hack hack... my $CRLF = "\015\012"; # "\r\n" is not portable my ( $data, $message, $protocol ) = @_; while ($$data ne "" && grep({$_ ne ""} @{$self->{buffers}}) > 0) { $self->filter($data, $message, $protocol); if ( length $data ) { if ($chunked) { printf $conn "%x$CRLF%s$CRLF", length($data), $data; } else { print $conn $data; } } } # call the cleanup routine if needed for ( @{ $self->{current} } ) { $_->end if $_->can('end'); } # clean up the mess for next time $self->eod; }
RT-Send-CC: book [...] cpan.org
[CDOLAN - Tue Jul 26 16:01:30 2005]: Show quoted text
> And here's a working version (attached). It rather hackish, primarily > in its determination of $chunked, but functional. >
Sorry for the delayed answer, and thanks for the request and the patch. This is an interesting request, and I see how this feature could be useful. I'm very short on time on the moment (probably until september), so don't expect this to appear soon in HTTP::Proxy. Version 0.16 should be out before YAPC::Europe (end of august), but it includes some other changes. I put your request on my TODO list, and will try to find a way to implement this.
From: cdolan [...] cpan.org
[BOOK - Sun Jul 31 17:42:13 2005]: Show quoted text
> Sorry for the delayed answer, and thanks for the request and the > patch. > This is an interesting request, and I see how this feature could be > useful. > > I'm very short on time on the moment (probably until september), > so don't expect this to appear soon in HTTP::Proxy. Version 0.16 > should be out before YAPC::Europe (end of august), but it includes > some other changes. I put your request on my TODO list, and will > try to find a way to implement this.
Philippe, I've built a more robust patch to implement this feature against HTTP-Proxy v0.16. The attached .zip file contains four files: * chunked.patch - Encapsulated the "print $conn" statements: all "if ($chunked)" tests are now in one place * booleans.patch - Changed $chunked, $sent and $last into properties $self->chunked, $self->sent, $self->keepalive * filter_last.patch - Changed filter_last to emit incremental data each iteration. Changed the FilterStack constructor to take an HTTP::Proxy instance * overall.patch - All three of the above patches rolled into one The filter_last.patch is what I was really trying to accomplish, but to keep it from being too messy, I needed to encapsulate the output routines. That led to the chunked.patch and booleans.patch cleanup effort. The three patches must be applied in the order above, or overall.patch can be used by itself. -- Chris

Message body not shown because it is not plain text.

Hi Philippe, Just a reminder that this patch is still available. If you don't like it, I'd be happy to refactor. As soon as this patch goes in, I'm hoping to release HTTP::Proxy::Throttle to make use of it. Currently, that unreleased module has a bunch of hacks into the HTTP::Proxy namespace... Best wishes, Chris [CDOLAN - Tue Sep 13 10:34:09 2005]: Show quoted text
> Philippe, > > I've built a more robust patch to implement this feature against > HTTP-Proxy v0.16. The attached .zip file contains four files: > > * chunked.patch - Encapsulated the "print $conn" statements: > all "if ($chunked)" tests are now in one place > * booleans.patch - Changed $chunked, $sent and $last into properties > $self->chunked, $self->sent, $self->keepalive > * filter_last.patch - Changed filter_last to emit incremental data > each iteration. Changed the FilterStack > constructor to take an HTTP::Proxy instance > * overall.patch - All three of the above patches rolled into one > > > The filter_last.patch is what I was really trying to accomplish, but to > keep it from being too messy, I needed to encapsulate the output > routines. That led to the chunked.patch and booleans.patch cleanup > effort. The three patches must be applied in the order above, or > overall.patch can be used by itself. > -- Chris
From: cdolan [...] cpan.org
Hi Philippe, Should I update this patch for 0.18, or should we just close this bug as "REJECTED"? It's been about 8 months since I first started working on this... Chris
On Wed Apr 26 10:10:10 2006, CLOTHO wrote: Show quoted text
> Hi Philippe, > > Should I update this patch for 0.18, or should we just close this bug as > "REJECTED"? > > It's been about 8 months since I first started working on this... > > Chris
I'm sorry I haven't yet introduced your patch in HTTP::Proxy. Release 0.18 was a quick release to close a few small bugs. The upcoming release 0.19 (on its way to CPAN as I write this) closes a few more small bugs. Please send me an updated patch, and I promise I'll look at it and introduce it the next release (0.20). Thanks for your patience, Philippe
From: cdolan [...] cpan.org
Attached are my patches rebuilt against 0.19 instead of 0.16. Luckily, there was only one conflict, so it was easy. Included are the three individual patches and one overall patch that is the previous three combined. With this patch, one trivial test fails: a pod-coverage test for the three boolean accessors I added. Thanks for this very useful module. Chris

Message body not shown because it is not plain text.