Skip Menu |

This queue is for tickets about the Geo-GDAL CPAN distribution.

Report information
The Basics
Id: 120531
Status: resolved
Priority: 0/
Queue: Geo-GDAL

People
Owner: ari.jolma [...] gmail.com
Requestors: pliggatt [...] microdecisions.com
Cc:
AdminCc:

Bug Information
Severity: (no value)
Broken in: (no value)
Fixed in: (no value)



Subject: GDAL.pm processes parameters with more than one argument using a hash reference in an unexpected way
Date: Mon, 6 Mar 2017 20:41:34 +0000
To: "bug-Geo-GDAL [...] rt.cpan.org" <bug-Geo-GDAL [...] rt.cpan.org>
From: "Dr. Peter A. J. Liggatt" <pliggatt [...] microdecisions.com>
We are running perl 5.16.3 built for x86_64-linux-thread-multi on linux 3.10.0-327.10.1.el7.x86_64 In GDAL.pm version 2.0.12 built by SWIG during a GDAL source build, there is the following subroutine at line 1402: sub make_processing_options { my ($o) = @_; if (ref $o eq 'HASH') { for my $key (keys %$o) { unless ($key =~ /^-/) { $o->{'-'.$key} = $o->{$key}; delete $o->{$key}; } } $o = [%$o]; } return $o; } The highlighted line will correctly convert arguments such as -of => 'gtiff', '-r' => 'Lanczos' as expected. However, for example, gdalwarp accepts parameters such as -te that have more than one argument. The documentation does describe that a list reference such as ['-tr', 10, 10] can be used, but that path through the code will not add a leading hyphen to, say, ['tr', 10, 10], and the hash reference / named parameter form seems to be a cleaner approach when passing many parameters. More so, this seems to be a trap for the unwary who try, say, '-ts' => '2048 2048' to have it fail mysteriously. After all, the documentation does not explicitly exclude that syntax and the error from gdalwarp is that the parameter is unknown rather than it has the wrong number of arguments. I would suggest that the highlighted line be recoded to be DWIM - the written construct is unusual and opaque - so that the syntax 'ts' => '2048 2048' succeeds, or at least the documentation be clarified that the hash reference form does not work for multiple arguments. The following line would appear to do that, but this is also opaque and it might be better to review the for loop instead. $o = [ map { $_, split(/\s+/, $o->{$_}) } keys %$o ]; Dr. Peter A. J. Liggatt President MicroDecisions, Inc. Tel: 407-843-1550 E-mail: pliggatt@microdecisions.com<mailto:pliggatt@microdecisions.com> Website: www.microbase.com<http://www.microbase.com/>
On ma 06.maalis 2017 18:15:49, pliggatt@microdecisions.com wrote: Show quoted text
> We are running perl 5.16.3 built for x86_64-linux-thread-multi on > linux 3.10.0-327.10.1.el7.x86_64 > > In GDAL.pm version 2.0.12 built by SWIG during a GDAL source build, > there is the following subroutine at line 1402:
That must be GDAL 2.1, i.e, module version >=2.01 since make_processing_options appeared in the bindings when GDAL was in 2.1. Show quoted text
> > sub make_processing_options { > my ($o) = @_; > if (ref $o eq 'HASH') { > for my $key (keys %$o) { > unless ($key =~ /^-/) { > $o->{'-'.$key} = $o->{$key}; > delete $o->{$key}; > } > } > $o = [%$o]; > } > return $o; > } > > The highlighted line will correctly convert arguments such as > > -of => 'gtiff', '-r' => 'Lanczos' > > as expected. However, for example, gdalwarp accepts parameters such as > -te that have more than one argument. The documentation does describe > that a list reference such as ['-tr', 10, 10] can be used, but that > path through the code will not add a leading hyphen to, say, ['tr', > 10, 10], and the hash reference / named parameter form seems to be a > cleaner approach when passing many parameters. More so, this seems to > be a trap for the unwary who try, say, '-ts' => '2048 2048' to have > it fail mysteriously. After all, the documentation does not explicitly > exclude that syntax and the error from gdalwarp is that the parameter > is unknown rather than it has the wrong number of arguments. > > I would suggest that the highlighted line be recoded to be DWIM - the > written construct is unusual and opaque - so that the syntax 'ts' => > '2048 2048' succeeds, or at least the documentation be clarified that > the hash reference form does not work for multiple arguments.
The alternative to give the options without the hyphen '-' was basically a mistake as I thought (wrongly) that {-ts=>[]} (i.e., no need for '-ts') is not allowed (it is). I think the correct way would be to remove the hyphenless form from the documentation. (But leave it in the code) Show quoted text
> > The following line would appear to do that, but this is also opaque > and it might be better to review the for loop instead. > > $o = [ map { $_, split(/\s+/, $o->{$_}) } keys %$o ];
I would add to the loop this $val = [split /\s+/, $val] if $val =~ /\s+/; This assumes that there are and not will be option values in GDAL command line tools that may contain whitespace. I'm not (yet) sure about that - I'll check, but of course future we can't know. Note: the above line sits into the new version (2.2) of that sub better than in the old one (2.1), which does not have $val. Thanks for the report, Ari Show quoted text
> > > Dr. Peter A. J. Liggatt > President > MicroDecisions, Inc. > Tel: 407-843-1550 > E-mail: > pliggatt@microdecisions.com<mailto:pliggatt@microdecisions.com> > Website: www.microbase.com<http://www.microbase.com/>
Fixed in https://trac.osgeo.org/gdal/changeset/37630 Note that I added support for option values that have whitespace but do not make up a list. I did not change older versions or documentation. The new documentation will be at http://arijolma.org/Geo-GDAL/snapshot/ tomorrow. Ari On Tue 07.maalis 2017 02:21:23, AJOLMA wrote: Show quoted text
> On ma 06.maalis 2017 18:15:49, pliggatt@microdecisions.com wrote:
> > We are running perl 5.16.3 built for x86_64-linux-thread-multi on > > linux 3.10.0-327.10.1.el7.x86_64 > > > > In GDAL.pm version 2.0.12 built by SWIG during a GDAL source build, > > there is the following subroutine at line 1402:
> > That must be GDAL 2.1, i.e, module version >=2.01 since > make_processing_options > appeared in the bindings when GDAL was in 2.1. >
> > > > sub make_processing_options { > > my ($o) = @_; > > if (ref $o eq 'HASH') { > > for my $key (keys %$o) { > > unless ($key =~ /^-/) { > > $o->{'-'.$key} = $o->{$key}; > > delete $o->{$key}; > > } > > } > > $o = [%$o]; > > } > > return $o; > > } > > > > The highlighted line will correctly convert arguments such as > > > > -of => 'gtiff', '-r' => 'Lanczos' > > > > as expected. However, for example, gdalwarp accepts parameters such > > as > > -te that have more than one argument. The documentation does describe > > that a list reference such as ['-tr', 10, 10] can be used, but that > > path through the code will not add a leading hyphen to, say, ['tr', > > 10, 10], and the hash reference / named parameter form seems to be a > > cleaner approach when passing many parameters. More so, this seems to > > be a trap for the unwary who try, say, '-ts' => '2048 2048' to have > > it fail mysteriously. After all, the documentation does not > > explicitly > > exclude that syntax and the error from gdalwarp is that the parameter > > is unknown rather than it has the wrong number of arguments. > > > > I would suggest that the highlighted line be recoded to be DWIM - the > > written construct is unusual and opaque - so that the syntax 'ts' => > > '2048 2048' succeeds, or at least the documentation be clarified that > > the hash reference form does not work for multiple arguments.
> > The alternative to give the options without the hyphen '-' was > basically > a mistake as I thought (wrongly) that {-ts=>[]} (i.e., no need for '- > ts') > is not allowed (it is). > > I think the correct way would be to remove the hyphenless form from > the > documentation. (But leave it in the code) >
> > > > The following line would appear to do that, but this is also opaque > > and it might be better to review the for loop instead. > > > > $o = [ map { $_, split(/\s+/, $o->{$_}) } keys %$o ];
> > I would add to the loop this > > $val = [split /\s+/, $val] if $val =~ /\s+/; > > This assumes that there are and not will be option values in GDAL > command line > tools that may contain whitespace. I'm not (yet) sure about that - > I'll check, but > of course future we can't know. > > Note: the above line sits into the new version (2.2) of that sub > better than in > the old one (2.1), which does not have $val. > > Thanks for the report, > > Ari >
> > > > > > Dr. Peter A. J. Liggatt > > President > > MicroDecisions, Inc. > > Tel: 407-843-1550 > > E-mail: > > pliggatt@microdecisions.com<mailto:pliggatt@microdecisions.com> > > Website: www.microbase.com<http://www.microbase.com/>
Subject: RE: [rt.cpan.org #120531] Resolved: GDAL.pm processes parameters with more than one argument using a hash reference in an unexpected way
Date: Wed, 8 Mar 2017 19:19:41 +0000
To: "bug-Geo-GDAL [...] rt.cpan.org" <bug-Geo-GDAL [...] rt.cpan.org>
From: "Dr. Peter A. J. Liggatt" <pliggatt [...] microdecisions.com>
Thank you! Dr. Peter A. J. Liggatt President MicroDecisions, Inc. Tel:  407-843-1550 E-mail: pliggatt@microdecisions.com Website: www.microbase.com Show quoted text
-----Original Message----- From: Ari Jolma via RT [mailto:bug-Geo-GDAL@rt.cpan.org] Sent: Tuesday, March 07, 2017 3:33 AM To: Dr. Peter A. J. Liggatt <pliggatt@microdecisions.com> Subject: [rt.cpan.org #120531] Resolved: GDAL.pm processes parameters with more than one argument using a hash reference in an unexpected way <URL: https://rt.cpan.org/Ticket/Display.html?id=120531 > According to our records, your request has been resolved. If you have any further questions or concerns, please respond to this message.
Subject: RE: [rt.cpan.org #120531] GDAL.pm processes parameters with more than one argument using a hash reference in an unexpected way
Date: Wed, 8 Mar 2017 19:58:56 +0000
To: "bug-Geo-GDAL [...] rt.cpan.org" <bug-Geo-GDAL [...] rt.cpan.org>
From: "Dr. Peter A. J. Liggatt" <pliggatt [...] microdecisions.com>
By the way, yes, it is version 2.1.2. I may be forgiven for reading the top two lines of GDAL.pm that reads as below. Presumably that is the SWIG version. # This file was automatically generated by SWIG (http://www.swig.org). # Version 2.0.12 and did not read down to our $VERSION = '2.0102'; our $GDAL_VERSION = '2.1.2'; Dr. Peter A. J. Liggatt President MicroDecisions, Inc. Tel:  407-843-1550 E-mail: pliggatt@microdecisions.com Website: www.microbase.com Show quoted text
-----Original Message----- From: Ari Jolma via RT [mailto:bug-Geo-GDAL@rt.cpan.org] Sent: Tuesday, March 07, 2017 2:21 AM To: Dr. Peter A. J. Liggatt <pliggatt@microdecisions.com> Subject: [rt.cpan.org #120531] GDAL.pm processes parameters with more than one argument using a hash reference in an unexpected way <URL: https://rt.cpan.org/Ticket/Display.html?id=120531 > On ma 06.maalis 2017 18:15:49, pliggatt@microdecisions.com wrote:
> We are running perl 5.16.3 built for x86_64-linux-thread-multi on > linux 3.10.0-327.10.1.el7.x86_64 > > In GDAL.pm version 2.0.12 built by SWIG during a GDAL source build, > there is the following subroutine at line 1402:
That must be GDAL 2.1, i.e, module version >=2.01 since make_processing_options appeared in the bindings when GDAL was in 2.1.
> > sub make_processing_options { > my ($o) = @_; > if (ref $o eq 'HASH') { > for my $key (keys %$o) { > unless ($key =~ /^-/) { > $o->{'-'.$key} = $o->{$key}; > delete $o->{$key}; > } > } > $o = [%$o]; > } > return $o; > } > > The highlighted line will correctly convert arguments such as > > -of => 'gtiff', '-r' => 'Lanczos' > > as expected. However, for example, gdalwarp accepts parameters such as > -te that have more than one argument. The documentation does describe > that a list reference such as ['-tr', 10, 10] can be used, but that > path through the code will not add a leading hyphen to, say, ['tr', > 10, 10], and the hash reference / named parameter form seems to be a > cleaner approach when passing many parameters. More so, this seems to > be a trap for the unwary who try, say, '-ts' => '2048 2048' to have > it fail mysteriously. After all, the documentation does not explicitly > exclude that syntax and the error from gdalwarp is that the parameter > is unknown rather than it has the wrong number of arguments. > > I would suggest that the highlighted line be recoded to be DWIM - the > written construct is unusual and opaque - so that the syntax 'ts' => > '2048 2048' succeeds, or at least the documentation be clarified that > the hash reference form does not work for multiple arguments.
The alternative to give the options without the hyphen '-' was basically a mistake as I thought (wrongly) that {-ts=>[]} (i.e., no need for '-ts') is not allowed (it is). I think the correct way would be to remove the hyphenless form from the documentation. (But leave it in the code)
> > The following line would appear to do that, but this is also opaque > and it might be better to review the for loop instead. > > $o = [ map { $_, split(/\s+/, $o->{$_}) } keys %$o ];
I would add to the loop this $val = [split /\s+/, $val] if $val =~ /\s+/; This assumes that there are and not will be option values in GDAL command line tools that may contain whitespace. I'm not (yet) sure about that - I'll check, but of course future we can't know. Note: the above line sits into the new version (2.2) of that sub better than in the old one (2.1), which does not have $val. Thanks for the report, Ari
> > > Dr. Peter A. J. Liggatt > President > MicroDecisions, Inc. > Tel: 407-843-1550 > E-mail: > pliggatt@microdecisions.com<mailto:pliggatt@microdecisions.com> > Website: www.microbase.com<http://www.microbase.com/>
Subject: RE: [rt.cpan.org #120531] GDAL.pm processes parameters with more than one argument using a hash reference in an unexpected way
Date: Wed, 8 Mar 2017 19:58:56 +0000
To: "bug-Geo-GDAL [...] rt.cpan.org" <bug-Geo-GDAL [...] rt.cpan.org>
From: "Dr. Peter A. J. Liggatt" <pliggatt [...] microdecisions.com>
By the way, yes, it is version 2.1.2. I may be forgiven for reading the top two lines of GDAL.pm that reads as below. Presumably that is the SWIG version. # This file was automatically generated by SWIG (http://www.swig.org). # Version 2.0.12 and did not read down to our $VERSION = '2.0102'; our $GDAL_VERSION = '2.1.2'; Dr. Peter A. J. Liggatt President MicroDecisions, Inc. Tel:  407-843-1550 E-mail: pliggatt@microdecisions.com Website: www.microbase.com Show quoted text
-----Original Message----- From: Ari Jolma via RT [mailto:bug-Geo-GDAL@rt.cpan.org] Sent: Tuesday, March 07, 2017 2:21 AM To: Dr. Peter A. J. Liggatt <pliggatt@microdecisions.com> Subject: [rt.cpan.org #120531] GDAL.pm processes parameters with more than one argument using a hash reference in an unexpected way <URL: https://rt.cpan.org/Ticket/Display.html?id=120531 > On ma 06.maalis 2017 18:15:49, pliggatt@microdecisions.com wrote:
> We are running perl 5.16.3 built for x86_64-linux-thread-multi on > linux 3.10.0-327.10.1.el7.x86_64 > > In GDAL.pm version 2.0.12 built by SWIG during a GDAL source build, > there is the following subroutine at line 1402:
That must be GDAL 2.1, i.e, module version >=2.01 since make_processing_options appeared in the bindings when GDAL was in 2.1.
> > sub make_processing_options { > my ($o) = @_; > if (ref $o eq 'HASH') { > for my $key (keys %$o) { > unless ($key =~ /^-/) { > $o->{'-'.$key} = $o->{$key}; > delete $o->{$key}; > } > } > $o = [%$o]; > } > return $o; > } > > The highlighted line will correctly convert arguments such as > > -of => 'gtiff', '-r' => 'Lanczos' > > as expected. However, for example, gdalwarp accepts parameters such as > -te that have more than one argument. The documentation does describe > that a list reference such as ['-tr', 10, 10] can be used, but that > path through the code will not add a leading hyphen to, say, ['tr', > 10, 10], and the hash reference / named parameter form seems to be a > cleaner approach when passing many parameters. More so, this seems to > be a trap for the unwary who try, say, '-ts' => '2048 2048' to have > it fail mysteriously. After all, the documentation does not explicitly > exclude that syntax and the error from gdalwarp is that the parameter > is unknown rather than it has the wrong number of arguments. > > I would suggest that the highlighted line be recoded to be DWIM - the > written construct is unusual and opaque - so that the syntax 'ts' => > '2048 2048' succeeds, or at least the documentation be clarified that > the hash reference form does not work for multiple arguments.
The alternative to give the options without the hyphen '-' was basically a mistake as I thought (wrongly) that {-ts=>[]} (i.e., no need for '-ts') is not allowed (it is). I think the correct way would be to remove the hyphenless form from the documentation. (But leave it in the code)
> > The following line would appear to do that, but this is also opaque > and it might be better to review the for loop instead. > > $o = [ map { $_, split(/\s+/, $o->{$_}) } keys %$o ];
I would add to the loop this $val = [split /\s+/, $val] if $val =~ /\s+/; This assumes that there are and not will be option values in GDAL command line tools that may contain whitespace. I'm not (yet) sure about that - I'll check, but of course future we can't know. Note: the above line sits into the new version (2.2) of that sub better than in the old one (2.1), which does not have $val. Thanks for the report, Ari
> > > Dr. Peter A. J. Liggatt > President > MicroDecisions, Inc. > Tel: 407-843-1550 > E-mail: > pliggatt@microdecisions.com<mailto:pliggatt@microdecisions.com> > Website: www.microbase.com<http://www.microbase.com/>
The change that I made to the trunk introduces a backwards incompatibility for -sql option. Before $dataset->Rasterize([-sql = $sql]); worked but now it needs to be written -sql = "'$sql'" or -sql = [$sql]. It really seems to be that for some options like -te it is natural to expect split by space but that is not the case for -sql. Furthermore, if options are given in an array, the option key is not available and we can't write an explicit test for -sql. As a compromise I'll change the make_processing_options code to split only when the option value is a list of numbers.