Bug #67336 for MongoDB: looks_like

Fri Apr 08 15:12:15 2011 jettero [...] cpan.org - Ticket created

Subject:

looks_like_number ... no?

MongoDB has some type checking that I'm unaccustomed to in Perl settings and I find counter-intuitive and arguably even wrong ... Although your opinion may differ from mine ... Consider this code: if( my $b = $cgi->param("b") ) { if( my $es = eval{ UnixDate(ParseDate($b), '%s') } ) { $search->{_id}{'$gte'} = $es; } } This was not working right at all until I had the idea that perhaps the type checking was happening on in C where you can more easily check things perl people sometimes cannot (ie, NV vs PV). Up at the high level, the best I can do is Scalar::util::looks_like_number($epoch_seconds_string); and it surely does. So I had the genius idea to do this: $search->{_id}{'$gte'} = 0 + $es; But I don't think I should have to, not in Perl anyway. In Python, sure, but not in Perl. -Paul -- If riding in an airplane is flying, then riding in a boat is swimming. 116 jumps, 48.6 minutes of freefall, 92.9 freefall miles.

Fri Apr 08 18:40:24 2011 jettero [...] cpan.org - Correspondence added

Show quoted text

> things perl people sometimes cannot (ie, NV vs PV).

IV, not NV. :) -- If riding in an airplane is flying, then riding in a boat is swimming. 116 jumps, 48.6 minutes of freefall, 92.9 freefall miles.

Fri Apr 08 18:40:25 2011 jettero [...] cpan.org - Status changed from 'new' to 'open'

Fri Apr 08 18:44:00 2011 KRISTINA [...] cpan.org - Correspondence added

What version of Perl are you using?

Fri Apr 08 22:13:01 2011 jettero [...] cpan.org - Correspondence added

On Fri Apr 08 18:44:00 2011, KRISTINA wrote: Show quoted text

> What version of Perl are you using?

This is perl, v5.10.1 (*) built for i686-linux-gnu-thread-multi I believe the same result would happen in nearly all Perls though. The problem (if you agree there is one) is in append_av in the .xs. where it cases on SVt_IV: or SVt_PV: and where it should (in my opinion) not rely an a number always being IV and a string always being PV. In fact they change around based on where they came from and what they were used for last. I assume all this is very obvious, and I'm aware I'm nit picking. -Paul -- If riding in an airplane is flying, then riding in a boat is swimming. 116 jumps, 48.6 minutes of freefall, 92.9 freefall miles.

Mon Apr 11 11:50:11 2011 KRISTINA [...] cpan.org - Correspondence added

Sorry, I misunderstood your question. The problem is that you might want to save a string that looks like a number, so how could you do that if the driver converted everything that looked like a number? The way the driver works now is that, if Perl thinks something might be a number or a string (PVIVs), it'll try to figure out which is more likely. If Perl says "this is a number" (IV) or "this is a string" (PV), the driver can't do much to figure out which the user meant.

Mon Apr 11 13:04:19 2011 jettero [...] cpan.org - Correspondence added

I can't argue that logic, but I still don't like it. I'll just have to get used to it. I'm too late to talk about the user interface. You couldn't possibly change it now. But, I'd rather see some way to explicitly talk about the types rather than having to hint around them. The hints should dwim (of course, what did I mean if there's no schema) and then if you need something else you should be able to say what you want. I don't like that I have load and inflict Math::BigInt just to do a 64 bit number if my number happens to be 3_000_000; and I don't like that "3" is stored as a string. What I'd rather I suppose is {'$set'=>{val=>{'i64'=>"3_000_000"}}} … Well, anyway, I suppose this ticket should be marked closed/invalid. Thanks! -Paul -- If riding in an airplane is flying, then riding in a boat is swimming. 116 jumps, 48.6 minutes of freefall, 92.9 freefall miles.

Mon Apr 11 13:09:27 2011 rafl [...] debian.org - Correspondence added

I'm completely with the requestor here. The unreliable and very much non-obvious hacks one has to pull on the perl side to get one's data stored in the right way are not cool and the biggest annoyance I have with using MongoDB from perl. Type annotations in the call site indeed seem to be the way to go. The current DWIMmy behaviour doesn't need to be changed, only a more explicit interface needs to be added.

Mon Jul 11 11:33:53 2011 sewi [...] cpan.org - Correspondence added

The problem is still open and it's critical (took me half a working day to find it). Perl people expect modules and drivers to identify numbers. Use this code to replay the error: ->query({ date => { '$gt' => "1310397110" } }) Usually, numbers are not written in quotes, but if a variable is used instead of a fixed number, noone knows how it was created. My value came from the CGI module parsing incoming data of a HTTP request. Perl people still see this as a number, whatever MongoDB::write_query is doing, it's wrong. If you don't want to fix it, at least issue a warning if a number operation ($gt, $lt, etc.) is being used with a string argument.

Mon Jul 11 15:22:41 2011 jettero [...] cpan.org - Correspondence added

On Mon Jul 11 11:33:53 2011, SEWI wrote: Show quoted text

> Usually, numbers are not written in quotes, but if a variable is used > instead of a fixed number, noone knows how it was created.

In fact, you can't actually tell from inside Perl without some unusual contortions. The best you can do is http://search.cpan.org/perldoc?Scalar::Util#looks_like_number Regular Perl people probably don't know the difference between an IV, a PV, and a PvIV and even if they do, they shouldn't have to. That's not how Perl works. If I wasn't so deep into this project and married to MongoDB, I'd have ditched it completely after the 5 hours that went into this tragedy: if( ref($value)eq 'SCALAR' ) { # NOTE: occasionally $value comes in as a PVIV instead of an SV = IV => # SV = PV; not totally sure what the difference is, but MongoDB does # the wrong thing by checking SVt_PV in perl_mongo.c $value = do { my $x = "$$value"; \$x }; # use Devel::Peek; # Dump($value); # Dump(\"goal"); # exit 0; } In short, it's not just the numerical handling. The MongoDB driver tries to make assumptions about what PV and IV mean and Perl's temporary internal format is the wrong way to check it. Use the context for that information. If the operation is numeric then "" is 0 and if the operation is a string operation then 33 is a string. That's how Perl works. -- If riding in an airplane is flying, then riding in a boat is swimming. 116 jumps, 48.6 minutes of freefall, 92.9 freefall miles.

Mon Jul 11 17:02:45 2011 KRISTINA [...] cpan.org - Correspondence added

Okay, I understand that everyone wants this to be easier and it is very important. So, to not break backwards compat (although it's pissing people off so much I'm not sure it isn't worth breaking bc...) here's my plan: * make the driver treat anything that looks_like_number as a number. * add a flag to choose old-style (attempt to auto-detect numbers) vs. new-style (attempt to save anything that looks like a number as a number). This is a pretty trivial change, the only issue is: Some things look like numbers but shouldn't be saved as numbers (e.g., zip codes). To handle these, there some sort of Mongo::String type or annotation for forcing the driver to save them as strings. I don't particularly like adding a string class, so: *do people have any suggestions on a Perl-ish way of doing this?*

Mon Jul 11 21:57:03 2011 jettero [...] cpan.org - Correspondence added

On Mon Jul 11 17:02:45 2011, KRISTINA wrote: Show quoted text

> So, to not break backwards compat (although it's pissing people off so > much I'm not sure it isn't worth breaking bc...)

I can't speak for anyone else, but I was never pissed off. Frustrated, sure, but never angry. The work you and the rest of the Mongodb people are doing is fun and exciting and all this stuff is new. New stuff has little problems. And frustrating though this stuff is, it's not that bad in the grand scheme of things. It just hasn't been all worked out yet. -- If riding in an airplane is flying, then riding in a boat is swimming. 116 jumps, 48.6 minutes of freefall, 92.9 freefall miles.

Tue Jul 12 03:01:48 2011 sewi [...] cpan.org - Correspondence added

Am Mo 11. Jul 2011, 17:02:45, KRISTINA schrieb: [...] Show quoted text

> Some things look like numbers but shouldn't be saved as numbers (e.g., > zip codes). To handle these, there some sort of Mongo::String type or > annotation for forcing the driver to save them as strings. > > I don't particularly like adding a string class, so: *do people have

any Show quoted text

> suggestions on a Perl-ish way of doing this?*

Thank you for facing this! Some ideas: * DBIx::Class is using SCALAR-References for "raw" things, so $x = \'010' would be ref($x) eq 'SCALAR' and ${$x} giving the "real value". Drawback: References are not allowed in Hash keys. * A "custom" MongoDB modifier $x = {'$txt' => '010'} could tell the driver that the value is forced to be text (maybe use $bin instead), but may collide with further MongoDB operators. * A Datatype module like you did it for the OIDs $x = MongoDB::Datatype::Text->new(value => '010') would make the job. Not as simple as the others, but would be ok for the few cases where it's required.

Thu Jul 21 17:15:19 2011 KRISTINA [...] cpan.org - Correspondence added

I've added an option for aggressively parsing variables as numbers. If you'd like to try it out, it's at https://github.com/mongodb/mongo-perl-driver. How to use: $MongoDB::BSON::looks_like_number = 1; # defaults to 0 $collection->insert({myNum => "4"}); # inserts the number 4 On Tue Jul 12 03:01:48 2011, SEWI wrote: Show quoted text

> Am Mo 11. Jul 2011, 17:02:45, KRISTINA schrieb: > [...]

> > Some things look like numbers but shouldn't be saved as numbers (e.g., > > zip codes). To handle these, there some sort of Mongo::String type or > > annotation for forcing the driver to save them as strings. > > > > I don't particularly like adding a string class, so: *do people have

> any

> > suggestions on a Perl-ish way of doing this?*

> > Thank you for facing this! > > Some ideas: > * DBIx::Class is using SCALAR-References for "raw" things, so $x = > \'010' would be ref($x) eq 'SCALAR' and ${$x} giving the "real value". > Drawback: References are not allowed in Hash keys. > * A "custom" MongoDB modifier $x = {'$txt' => '010'} could tell the > driver that the value is forced to be text (maybe use $bin instead), > but may collide with further MongoDB operators. > * A Datatype module like you did it for the OIDs $x = > MongoDB::Datatype::Text->new(value => '010') would make the job. Not as > simple as the others, but would be ok for the few cases where it's > required.

Thu Jul 21 18:13:49 2011 KRISTINA [...] cpan.org - Correspondence added

...and added MongoDB::BSON::String type to force number-looking things to be saved as strings. See the DataTypes pod for examples of usage. Let me know any feedback, these will be in the 0.44 release. On Thu Jul 21 17:15:19 2011, KRISTINA wrote: Show quoted text

> I've added an option for aggressively parsing variables as numbers. If > you'd like to try it out, it's at > https://github.com/mongodb/mongo-perl-driver. > > How to use: > > $MongoDB::BSON::looks_like_number = 1; # defaults to 0 > $collection->insert({myNum => "4"}); # inserts the number 4 > > > On Tue Jul 12 03:01:48 2011, SEWI wrote:

> > Am Mo 11. Jul 2011, 17:02:45, KRISTINA schrieb: > > [...]

> > > Some things look like numbers but shouldn't be saved as numbers (e.g., > > > zip codes). To handle these, there some sort of Mongo::String type or > > > annotation for forcing the driver to save them as strings. > > > > > > I don't particularly like adding a string class, so: *do people have

> > any

> > > suggestions on a Perl-ish way of doing this?*

> > > > Thank you for facing this! > > > > Some ideas: > > * DBIx::Class is using SCALAR-References for "raw" things, so $x = > > \'010' would be ref($x) eq 'SCALAR' and ${$x} giving the "real value". > > Drawback: References are not allowed in Hash keys. > > * A "custom" MongoDB modifier $x = {'$txt' => '010'} could tell the > > driver that the value is forced to be text (maybe use $bin instead), > > but may collide with further MongoDB operators. > > * A Datatype module like you did it for the OIDs $x = > > MongoDB::Datatype::Text->new(value => '010') would make the job. Not as > > simple as the others, but would be ok for the few cases where it's > > required.

> >

Thu Jul 21 18:14:18 2011 KRISTINA [...] cpan.org - Status changed from 'open' to 'resolved'

Wed Aug 10 18:29:55 2011 ether [...] cpan.org - Severity Unimportant changed to Normal

Wed Aug 10 18:29:55 2011 ether [...] cpan.org - Fixed in 0.44 added

Bug #67336 for MongoDB: looks_like_number ... no?

Preferred bug tracker

Maintainer(s)' notes