Skip Menu |

This queue is for tickets about the Data-FormValidator CPAN distribution.

Maintainer(s)' notes

This is the bug queue for Data::FormValidator.

Report information
The Basics
Id: 24702
Status: resolved
Priority: 0/
Queue: Data-FormValidator

People
Owner: MARKSTOS [...] cpan.org
Requestors: GTERMARS [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Important
Broken in: (no value)
Fixed in: (no value)



Subject: DFV::Constraints::Upload processes input data, not filtered data
In Bug #22589 I added support for a new "get_filtered_data()" method, which allowed for constraints that operated on multiple fields to be able to access the -filtered- versions of other data fields. I missed DFV::Constraints::Upload, though, and it was still processing the original -input- data instead of the -filtered- data. This causes problems if you have an upload field that is both filtered+constrained, and for which the constraint expects that the filter has already been applied. The use case that I've run into that uncovered this was attempting to use DFV::Filters::Image::image_filter() (to restrict the resolution of uploaded images) and DFV::Constraints::Upload::file_max_bytes() (to restrict uploaded file size) at the same time. What I was expecting to see was that "image_filter" would filter and resize the image first, and then the resulting image would then be checked for max bytes using "file_max_bytes". However.... what I ended up with was that "file_max_bytes" would constrain against the original uploaded image, not the filtered/resized version (which may have been shrunk down far enough to be of acceptable size). I've attached a diff against DFV::Constraints::Upload from DFV v4.50, which updates it to use "get_filtered_data()" wherever possible. Test suites in DFV v4.50 all pass, although I will admit that I have -not- checked this patch against uploads when using Apache::Request or CGI::Simple.
Subject: dfv-upload-filtered-data.patch
Only in Data-FormValidator-4.50/: .Changes.swp diff -ru Data-FormValidator-4.50.orig/lib/Data/FormValidator/Constraints/Upload.pm Data-FormValidator-4.50/lib/Data/FormValidator/Constraints/Upload.pm --- Data-FormValidator-4.50.orig/lib/Data/FormValidator/Constraints/Upload.pm 2006-12-04 18:38:14.000000000 -0800 +++ Data-FormValidator-4.50/lib/Data/FormValidator/Constraints/Upload.pm 2007-01-31 12:58:13.741041627 -0800 @@ -79,10 +79,7 @@ # are no additional arguments"; # } - my $q = $self->get_input_data; - - $q->can('param') || - die 'file_format: data object missing param() method'; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; my $fh = _get_upload_fh($self); @@ -121,7 +118,7 @@ my @mt_exts = $t ? $t->extensions : (); ## setup filename to retrieve extension - my $fn = $q->param($field); + my $fn = $q->{$field}; my ($uploaded_ext) = ($fn =~ m/\.([\w\d]*)?$/); my $ext; @@ -160,7 +157,7 @@ ($max_width > 0) || die 'image_max_dimensions: maximum width must be > 0'; ($max_height > 0) || die 'image_max_dimensions: maximum height must be > 0'; - my $q = $self->get_input_data; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; my ($width,$height) = _get_img_size($self); @@ -193,9 +190,7 @@ $max_bytes = 1024*1024; # default to 1 Meg } - my $q = $self->get_input_data; - $q->can('param') || - die 'file_max_bytes: object missing param() method'; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; @@ -229,7 +224,7 @@ ($min_width > 0) || die 'image_min_dimensions: minimum width must be > 0'; ($min_height > 0) || die 'image_min_dimensions: minimum height must be > 0'; - my $q = $self->get_input_data; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; my ($width, $height) = _get_img_size($self); @@ -249,16 +244,13 @@ sub _get_img_size { my $self = shift; - my $q = $self->get_input_data; + my $q = $self->get_filtered_data; ## setup caller to make can errors more useful my $caller = (caller(1))[3]; my $pkg = __PACKAGE__ . "::"; $caller =~ s/$pkg//g; - $q->can('param') || die "$caller: data object missing param() method"; - $q->can('upload') || die "$caller: data object missing upload() method"; - my $field = $self->get_current_constraint_field; ## retrieve filehandle from query object. @@ -286,9 +278,16 @@ sub _get_upload_fh { my $self = shift; - my $q = $self->get_input_data; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; + ## Hash + if (ref($q) eq 'HASH') { + my $fh = $q->{$field}; + ## convert into seekable handle + return IO::File->new_from_fd(fileno($fh), "r"); + } + ## CGI::Simple object processing (slightly different from others) if ($q->isa('CGI::Simple')) { ## get filename @@ -338,6 +337,8 @@ } ## returns mime type if included as part of the send +## +## NOTE: retrieves from original uploaded, -UNFILTERED- data sub _get_upload_mime_type { my $self = shift; @@ -436,6 +437,11 @@ give up. The extension we return is based on the MIME type we found, rather than trusting the one that was uploaded. +B<NOTE:> if we have to fall back to using the MIME type provided by the +browser, we access it from the original I<input> data and not the +I<filtered> data. This should only cause issue when you have used a filter +to alter the type of file that was uploaded (e.g. image conversion). + =item file_max_bytes This function checks the maximum size of an uploaded file. By default,
From: MARKSTOS [...] cpan.org
I'd like to accept the patch described below, but it fails with CGI::Simple. I'm not currently using that either myself, but I would like to continue to support it. Would you mind looking into it? Otherwise we can ask on the list for other volunteers. Heres's what the failures look like: Mark # Running tests with CGI::Simple t/upload.t: can't get filehandle for field named does_not_exist_gif at lib/Data/FormValidator/Constraints/Upload.pm line 89. t/upload.t: can't get filehandle for field named 100x100_gif at lib/Data/FormValidator/Constraints/Upload.pm line 89. t/upload.t: can't get filehandle for field named hello_world at lib/Data/FormValidator/Constraints/Upload.pm line 89. ok 26 ok 27 - expect format failure ok 28 - expect non-existent failure not ok 29 - valid # Failed test 'valid' # in t/upload.t at line 143. not ok 30 - meta() returns hash ref # Failed test 'meta() returns hash ref' # in t/upload.t at line 146. # got: '' # expected: 'HASH' not ok 31 - setting extension meta data # Failed test 'setting extension meta data' # in t/upload.t at line 148. not ok 32 - setting mime_type meta data # Failed test 'setting mime_type meta data' # in t/upload.t at line 149. ok 33 - max_bytes Can't use an undefined value as a HASH reference at t/upload.t line 154. # Looks like you planned 50 tests but only ran 33. # Looks like you failed 4 tests of 33 run. # Looks like your test died just after 33. dubious Test returned status 255 (wstat 65280, 0xff00) DIED. FAILED tests 29-32, 34-50 Failed 21/50 tests, 58.00% okay Failed Test Stat Wstat Total Fail Failed List of Failed ------------------------------------------------------------------------------- t/upload.t 255 65280 50 38 76.00% 29-32 34-50 On Wed Jan 31 15:58:54 2007, GTERMARS wrote: Show quoted text
> In Bug #22589 I added support for a new "get_filtered_data()" method, > which allowed for constraints that operated on multiple fields to be > able to access the -filtered- versions of other data fields. > > I missed DFV::Constraints::Upload, though, and it was still processing > the original -input- data instead of the -filtered- data. > > This causes problems if you have an upload field that is both > filtered+constrained, and for which the constraint expects that the > filter has already been applied. The use case that I've run into that > uncovered this was attempting to use > DFV::Filters::Image::image_filter() (to restrict the resolution of > uploaded images) and DFV::Constraints::Upload::file_max_bytes() (to > restrict uploaded file size) at the same time. What I was expecting > to see was that "image_filter" would filter and resize the image > first, and then the resulting image would then be checked for max > bytes using "file_max_bytes". However.... what I ended up with was > that "file_max_bytes" would constrain against the original uploaded > image, not the filtered/resized version (which may have been shrunk > down far enough to be of acceptable size). > > I've attached a diff against DFV::Constraints::Upload from DFV v4.50, > which updates it to use "get_filtered_data()" wherever possible. Test > suites in DFV v4.50 all pass, although I will admit that I have -not- > checked this patch against uploads when using Apache::Request or > CGI::Simple.
From: GTERMARS [...] cpan.org
On Fri Jul 13 22:56:50 2007, MARKSTOS wrote: Show quoted text
> I'd like to accept the patch described below, but it fails with > CGI::Simple. I'm not currently using that either myself, but I would > like to continue to support it. Would you mind looking into it?
Mark, I've had another look at this and have reworked the patch so that it now works properly with all of the test suites (even the CGI::Simple ones). Notable differences from the previous version of this patch: 1) DFV::Results now -explicitly= grabs the filehandle for CGI::Simple uploads, rather than just the filename. CGI and Apache::Request both return a $fh if you call "$q->param($field)" for an upload field, while CGI::Simple just returns the filename. While the update here seems a bit quirky, there isn't a nice way that I can see that we can quickly query a CGI::Simple object to say "does -this- field contain a file upload?" 2) DFV::Constraints::Upload::_get_upload_fh() has been -dramatically- reduced. Knowing that we're always working with "filtered" data, we can expect that we've always got a HASH here (that's what the "filtered" data is). To accommodate the cases where CGI/Apache::Request objects have non-seekable filehandles, we turn -all- of the filehandles into something seekable. ---------- As with the previous version of this patch, this is against "DFV-4.50". I've tested it locally on my machine here with CGI-3.15 and CGI::Simple-0.077 and all of the test suites run through successfully.
diff -ru Data-FormValidator-4.50.orig/lib/Data/FormValidator/Constraints/Upload.pm Data-FormValidator-4.50/lib/Data/FormValidator/Constraints/Upload.pm --- Data-FormValidator-4.50.orig/lib/Data/FormValidator/Constraints/Upload.pm 2006-12-04 18:38:14.000000000 -0800 +++ Data-FormValidator-4.50/lib/Data/FormValidator/Constraints/Upload.pm 2007-08-27 14:47:43.000000000 -0700 @@ -78,11 +78,7 @@ # included 'params => []' in your constraint definition, even if there # are no additional arguments"; # } - - my $q = $self->get_input_data; - - $q->can('param') || - die 'file_format: data object missing param() method'; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; my $fh = _get_upload_fh($self); @@ -121,7 +117,7 @@ my @mt_exts = $t ? $t->extensions : (); ## setup filename to retrieve extension - my $fn = $q->param($field); + my $fn = $self->get_input_data->param($field); my ($uploaded_ext) = ($fn =~ m/\.([\w\d]*)?$/); my $ext; @@ -160,7 +156,7 @@ ($max_width > 0) || die 'image_max_dimensions: maximum width must be > 0'; ($max_height > 0) || die 'image_max_dimensions: maximum height must be > 0'; - my $q = $self->get_input_data; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; my ($width,$height) = _get_img_size($self); @@ -193,9 +189,7 @@ $max_bytes = 1024*1024; # default to 1 Meg } - my $q = $self->get_input_data; - $q->can('param') || - die 'file_max_bytes: object missing param() method'; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; @@ -229,7 +223,7 @@ ($min_width > 0) || die 'image_min_dimensions: minimum width must be > 0'; ($min_height > 0) || die 'image_min_dimensions: minimum height must be > 0'; - my $q = $self->get_input_data; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; my ($width, $height) = _get_img_size($self); @@ -249,16 +243,13 @@ sub _get_img_size { my $self = shift; - my $q = $self->get_input_data; + my $q = $self->get_filtered_data; ## setup caller to make can errors more useful my $caller = (caller(1))[3]; my $pkg = __PACKAGE__ . "::"; $caller =~ s/$pkg//g; - $q->can('param') || die "$caller: data object missing param() method"; - $q->can('upload') || die "$caller: data object missing upload() method"; - my $field = $self->get_current_constraint_field; ## retrieve filehandle from query object. @@ -286,58 +277,19 @@ sub _get_upload_fh { my $self = shift; - my $q = $self->get_input_data; + my $q = $self->get_filtered_data; my $field = $self->get_current_constraint_field; - ## CGI::Simple object processing (slightly different from others) - if ($q->isa('CGI::Simple')) { - ## get filename - my $fn = $q->param($field); - if (!$fn) { - warn sprintf("Failed to locate filename '%s'", $q->cgi_error); - return undef; - } - - ## return filename - return $q->upload($fn); - } - - ## NOTE: Both Apache::Upload and CGI filehandles are not seekable - ## this causes issues with File::MMagic... - - ## Apache::Request object processing - if ($q->isa('Apache::Request')) { - use IO::File; - my $upload = $q->upload($field); ## return Apache::Upload - - ## error checking - warn "Failed to locate upload object" && return undef unless $upload; - - ## return filehandle - return IO::File->new_from_fd(fileno($upload->fh), "r"); - } - - - ## only CGI.pm just in case for weird subclasses - ## generic data object (or CGI), CGI.pm has incomplete fh's nice huh - if ($q->isa('CGI')) { - use IO::File; - my $fh = $q->upload($field); - - warn "Failed to load fh for $field" && return undef unless $fh; - - #my $tmpfile = $q->tmpFileName($q->param($field)) || return undef; - #return FileHandle->new($tmpfile); - - ## convert into seekable handle - return IO::File->new_from_fd(fileno($fh), "r"); - } - - ## not going to figure it out - return undef; + # convert the FH for the filtered data into a -seekable- handle; + # depending on whether we're using CGI::Simple, CGI, or Apache::Request + # we might not have something -seekable-. + use IO::File; + return IO::File->new_from_fd(fileno($q->{$field}), 'r'); } ## returns mime type if included as part of the send +## +## NOTE: retrieves from original uploaded, -UNFILTERED- data sub _get_upload_mime_type { my $self = shift; @@ -436,6 +388,11 @@ give up. The extension we return is based on the MIME type we found, rather than trusting the one that was uploaded. +B<NOTE:> if we have to fall back to using the MIME type provided by the +browser, we access it from the original I<input> data and not the +I<filtered> data. This should only cause issue when you have used a filter +to alter the type of file that was uploaded (e.g. image conversion). + =item file_max_bytes This function checks the maximum size of an uploaded file. By default, diff -ru Data-FormValidator-4.50.orig/lib/Data/FormValidator/Results.pm Data-FormValidator-4.50/lib/Data/FormValidator/Results.pm --- Data-FormValidator-4.50.orig/lib/Data/FormValidator/Results.pm 2006-12-04 18:38:14.000000000 -0800 +++ Data-FormValidator-4.50/lib/Data/FormValidator/Results.pm 2007-08-27 14:49:59.000000000 -0700 @@ -1047,7 +1047,21 @@ my %return; foreach my $k ($data->param()){ # we expect param to return an array if there are multiple values - my @v = $data->param($k); + my @v; + + # CGI::Simple requires us to call 'upload()' to get upload data, + # while CGI/Apache::Request return it on calling 'param()'. + # + # This seems quirky, but there isn't a way for us to easily check if + # "this field contains a file upload" or not. + if (UNIVERSAL::isa($data,'CGI::Simple')) { + @v = $data->upload($k) || $data->param($k); + } + else { + @v = $data->param($k); + } + + # we expect param to return an array if there are multiple values $return{$k} = scalar(@v)>1 ? \@v : $v[0]; } return %return;
Subject: Re: [rt.cpan.org #24702] DFV::Constraints::Upload processes input data, not filtered data
Date: Tue, 28 Aug 2007 11:12:57 -0400
To: bug-Data-FormValidator [...] rt.cpan.org
From: Mark Stosberg <mark [...] summersault.com>
Graham, Thanks for the patch. It generally looks good. I have a question about one change: On Monday 27 August 2007 18:31, Graham TerMarsch via RT wrote: Show quoted text
> - my $fn = $q->param($field); > + my $fn = $self->get_input_data->param($field);
Did you mean *filtered* there? It's about the only reference left in this area to the raw input data. If it's correct, perhaps this line could use a comment to explain the exception. I'll try to get this finally reviewed and released soon. Thanks again, Mark -- . . . . 1997-2007: Ten Years of Excellence. . . . . . Mark Stosberg Principal Developer mark@summersault.com Summersault, LLC 765-939-9301 ext 202 database driven websites . . . . . http://www.summersault.com/ . . . . . . . .
From: GTERMARS [...] cpan.org
On Tue Aug 28 11:14:07 2007, mark@summersault.com wrote: Show quoted text
> Thanks for the patch. It generally looks good. I have a question > about one change: > > On Monday 27 August 2007 18:31, Graham TerMarsch via RT wrote:
> > - my $fn = $q->param($field); > > + my $fn = $self->get_input_data->param($field);
> > Did you mean *filtered* there? It's about the only reference left in > this area to the raw input data. If it's correct, perhaps this line > could use a comment to explain the exception.
No, actually I -DO- mean "input data" there... Reason is... we're trying to get the file -name- for the file that was uploaded, not the actual file -handle-. Only way we can get the name of the uploaded file is to go back to the original input data and get it from there. You're right, though, this might warrant a short comment to indicate it. Maybe something like... # Need original uploaded filename, which is only available # in the original -input- data.
Thanks again for the patch. When I applied it, I got test failures in upload.t, as shown below. Would you mind investigating? Lately, some platforms are having similar failures in the same script: http://cpantesters.perl.org/show/Data-FormValidator.html#Data-FormValidator-4.52 Maybe that is related. I've spent about an hour today prepare other DFV patches for release and would appreciate help with this. Mark ##### t/upload....# Adding CGI::Simple tests # testing with CGI.pm version: 3.05 # testing with CGI::Simple version: 0.077 # Running tests with CGI t/upload....ok 25/50# Running tests with CGI::Simple t/upload....NOK 29 # Failed test 'valid' # in t/upload.t at line 143. t/upload....NOK 34 # Failed test 'setting bytes meta data' # in t/upload.t at line 154. t/upload.t: imgsize test failed at lib/Data/FormValidator/Constraints/Upload.pm line 183. t/upload.t: imgsize test failed at lib/Data/FormValidator/Constraints/Upload.pm line 183. t/upload....NOK 36 # Failed test 'expecting success with max_dimensions' # in t/upload.t at line 184. t/upload....ok 37/50Can't use an undefined value as a HASH reference at t/upload.t line 187. # Looks like you planned 50 tests but only ran 37. # Looks like you failed 3 tests of 37 run. # Looks like your test died just after 37. t/upload....dubious Test returned status 255 (wstat 65280, 0xff00) DIED. FAILED tests 29, 34, 36, 38-50 Failed 16/50 tests, 68.00% okay Failed Test Stat Wstat Total Fail Failed List of Failed ------------------------------------------------------------------------------- t/upload.t 255 65280 50 29 58.00% 29 34 36 38-50
From: GTERMARS [...] cpan.org
On Sat Oct 20 15:54:17 2007, MARKSTOS wrote: Show quoted text
> Thanks again for the patch. > > When I applied it, I got test failures in upload.t, as shown below. > Would you mind investigating?
I just had a run through, applying the patch to "4.52" and didn't have any troubles with it. That said, I'm quite possibly using a different set of supporting modules than others (I can't explain the difference immediately either): Perl 5.8.8 Test::More 0.62 CGI::Simple 0.077 (same version as you) I've also tested with: Test::More 0.72 CGI::Simple 1.103 and get the same results. Perhaps its more Perl (as in "core") related; which version are you running? There's gotta be something we're using different... :) Show quoted text
> Lately, some platforms are having similar failures in the same > script: > http://cpantesters.perl.org/show/Data-FormValidator.html#Data- > FormValidator-4.52 > > Maybe that is related.
I just checked on those failures and don't think they're related; the bulk of those failures have to do with a test being executed before the "plan" has been set up. I ran into this myself... you've got some "use_ok()" calls in a BEGIN block, but you do the "plan tests..." later on (after you've figured out how many tests you need to run). I -think- that something "Test::*" related got changed recently which is what causes those tests to now fail; they run fine for me on my dev box but I've had similar failures from other testers. Choices for fixing this are either (a) figure out how many tests you're going to run -in- the BEGIN block before the "use_ok()" calls, or (b) don't make them "use_ok()".
Subject: Re: [rt.cpan.org #24702] DFV::Constraints::Upload processes input data, not filtered data
Date: Sun, 21 Oct 2007 11:48:37 -0400
To: bug-Data-FormValidator [...] rt.cpan.org
From: Mark Stosberg <mark [...] summersault.com>
Graham, I figured this out, and got the test to pass. thanks for your help. The fix was patching your code like this: - return IO::File->new_from_fd(fileno($q->{$field}), 'r'); + + # If we we already have an IO::File object, return it, otherwise create one. + require Scalar::Util; + + if ( Scalar::Util::blessed($q->{$field}) && $q->{$field}->isa('IO::File') ) { + return $q->{$field}; + } + else { + return IO::File->new_from_fd(fileno($q->{$field}), 'r'); + } + #### I suppose that could mean that a newer version of IO::File was updated to allow your code to work. I also made another small refactor your code to avoid using "UNIVERSAL::can()", which another module will complain about. 4.55 is being sent to CPAN now. Mark