Bug #34822 for Math-BigInt: Which library is used?

Thu Apr 10 07:36:38 2008 MARKOV [...] cpan.org - Ticket created

Subject:

Which library is used?

My XML::Compile needs Math::BigInt. For performance reasons, I would like to include "try => GMP" everywhere, but that is not compatible with all current Linux distributions which include Perl-core modules. "lib => GMP" gives a warning, which is also not desired. Thinking about this, I cannot find in the documentation what happens when one module says "lib => GMP" and the other module some other library. Could I remove my explicit request for GMP, and tell the user to add it to the application script to get more performance?

Thu Apr 10 16:09:13 2008 nospam-abuse [...] bloodgate.com - Correspondence added

Subject:	Re: [rt.cpan.org #34822] Which library is used?
Date:	Thu, 10 Apr 2008 22:06:42 +0200
To:	bug-Math-BigInt [...] rt.cpan.org
From:	Tels <nospam-abuse [...] bloodgate.com>

Moin Mark, On Thursday 10 April 2008 13:36:44 Mark Overmeer via RT wrote: Show quoted text

> Thu Apr 10 07:36:38 2008: Request 34822 was acted upon. > Transaction: Ticket created by MARKOV > Queue: Math-BigInt > Subject: Which library is used? > Broken in: (no value) > Severity: Normal > Owner: Nobody > Requestors: MARKOV@cpan.org > Status: new > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=34822 > > > > My XML::Compile needs Math::BigInt. For performance reasons, I would > like to include "try => GMP" everywhere, but that is not compatible > with all current Linux distributions which include Perl-core modules. > "lib => GMP" gives a warning, which is also not desired. > > Thinking about this, I cannot find in the documentation what happens > when one module says "lib => GMP" and the other module some other > library. > > Could I remove my explicit request for GMP, and tell the user to add > it to the application script to get more performance?

The library that is loaded last, wins. Basically, any line of "use Math::BigInt lib => 'foo';" will try to load "Math::BigInt::foo" and if successfuly, use this library from this point on. The other libraries remain in memory, but are not used. Note that switching libraries _after_ constructing already Math::BigInt objects is a bad idea, as these objects cannot mix, e.g. if $a was constructed library GMP and $b with Calc, you cannot do "$a + $b" as this will blow up spectaculary :) For libraries that are used by other programs (e.g. you case) there are a few cases how to solve this: * just do "use Math::BigInt" and let the user decide which library to use underneath. (needs doc and education) * do like Math::String, use "try => GMP" and add a requirement for Math::BigInt v1.88. Which way is best depends on your situation and liking, tho. I agree the issue is tricky. Btw, _if_ your library constructs Math::BigInt (et. al) objects are startup, you can also use the same trick that Math::BigFloat does, in sub import do: # find out which one was actually loaded $MBI = Math::BigInt->config()->{lib}; # register us with MBI to get notified of future lib changes Math::BigInt::_register_callback( $self, sub { $MBI = $_[0]; } ); This registers a callback so that when someone does "use Math::BigInt lib => FOO;" behind your back, then $MBI will contain "Math::BigInt::FOO" afterwards. Of course, the callback would also need to "reconstruct" any already constructed objects. Hope that clears this up, I think it might be described somewhere in the documentation, but it is surely buried. Sorry that this is so complicated. Tels -- Signed on Thu Apr 10 21:54:46 2008 with key 0x93B84C15. Get one of my photo posters: http://bloodgate.com/posters PGP key on http://bloodgate.com/tels.asc or per email. "Und jetzt kommen Sie!" -- Stenkelfeld

Download (untitled)
application/pgp-signature 481b

Message body not shown because it is not plain text.

Thu Apr 10 16:09:39 2008 The RT System itself - Status changed from 'new' to 'open'

Thu Apr 10 17:05:17 2008 solutions [...] overmeer.net - Correspondence added

Subject:	Re: [rt.cpan.org #34822] Which library is used?
Date:	Thu, 10 Apr 2008 23:04:47 +0200
To:	"nospam-abuse [...] bloodgate.com via RT" <bug-Math-BigInt [...] rt.cpan.org>
From:	Mark Overmeer <solutions [...] overmeer.net>

Show quoted text

> <URL: http://rt.cpan.org/Ticket/Display.html?id=34822 > > Moin Mark,

Hi Tels, Show quoted text

> On Thursday 10 April 2008 13:36:44 Mark Overmeer via RT wrote:

> > Thu Apr 10 07:36:38 2008: Request 34822 was acted upon. > > Transaction: Ticket created by MARKOV > > Queue: Math-BigInt > > Subject: Which library is used? > > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=34822 >

Show quoted text

> The library that is loaded last, wins. Basically, any line of "use > Math::BigInt lib => 'foo';" will try to load "Math::BigInt::foo" and if > successfuly, use this library from this point on.

I understand the problems designing the interface, but it gives major problems, certainly when there is dynamic loading in play as well. Show quoted text

> Note that switching libraries _after_ constructing already Math::BigInt > objects is a bad idea, as these objects cannot mix, e.g. if $a was > constructed library GMP and $b with Calc, you cannot do "$a + $b" as > this will blow up spectaculary :)

You may freeze the library at INIT or CHECK type. Show quoted text

> For libraries that are used by other programs (e.g. you case) there are > a few cases how to solve this: > * just do "use Math::BigInt" and let the user decide which library to > use underneath. (needs doc and education)

But then the user needs to add "use Math::BigInt", what they do not expect. But I can try to describe it in the "performance" chapter of the docs. Show quoted text

> * do like Math::String, use "try => GMP" and add a requirement for > Math::BigInt v1.88.

The problems people report me are 1.87 in many Linux distributions, in combination with the problems with dual-lived modules... Show quoted text

> Hope that clears this up, I think it might be described somewhere in the > documentation, but it is surely buried. > Sorry that this is so complicated.

That's life ;-) -- Thanks for the swift reply, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net

Thu Apr 10 17:25:36 2008 nospam-abuse [...] bloodgate.com - Correspondence added

Subject:	Re: [rt.cpan.org #34822] Which library is used?
Date:	Thu, 10 Apr 2008 23:24:52 +0200
To:	bug-Math-BigInt [...] rt.cpan.org
From:	Tels <nospam-abuse [...] bloodgate.com>

Moin mark, On Thursday 10 April 2008 23:05:19 Mark Overmeer via RT wrote: Show quoted text

> Queue: Math-BigInt > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=34822 >

> > Moin Mark,

> Hi Tels,

> > The library that is loaded last, wins. Basically, any line of "use > > Math::BigInt lib => 'foo';" will try to load "Math::BigInt::foo" > > and if successfuly, use this library from this point on.

> > I understand the problems designing the interface, but it gives major > problems, certainly when there is dynamic loading in play as well.

Well, duh! :) The library interface was designed to allow changing the library instead of hardcoding it in - it wasn't thought at that time that there would be a change of library at run-time, or that this would actually cause the problems with objects that were constructed between loading of two different libraries - for the very simple reason that back than nobody even thought of these scenarios. (I clearly should have thought things more through, but well, I didn't :) Show quoted text

> > Note that switching libraries _after_ constructing already > > Math::BigInt objects is a bad idea, as these objects cannot mix, > > e.g. if $a was constructed library GMP and $b with Calc, you cannot > > do "$a + $b" as this will blow up spectaculary :)

> > You may freeze the library at INIT or CHECK type.

I may? Apart from the fact that I don't know how how to do this, I am also not sure that changing it now would actually work. Show quoted text

> > For libraries that are used by other programs (e.g. you case) there > > are a few cases how to solve this: > > * just do "use Math::BigInt" and let the user decide which library > > to use underneath. (needs doc and education)

> > But then the user needs to add "use Math::BigInt", what they do not > expect. But I can try to describe it in the "performance" chapter > of the docs.

Yes that is the drawback of this method. Show quoted text

> > * do like Math::String, use "try => GMP" and add a requirement for > > Math::BigInt v1.88.

> > The problems people report me are 1.87 in many Linux distributions, > in combination with the problems with dual-lived modules...

This is squarely my fault because I didn't release v1.88 of Math::BigInt about half a year ago.2007-01-27 v1.78 Tels * implement "try" and "only" as replacements for "lib" However, the "try" and "only" keywords were added in v1.78 of Math::BigInt, which is a bit older: 2007-01-27 v1.78 Tels * implement "try" and "only" as replacements for "lib" But I guess with the slow-moving target of distributions, that this is in still quite a lot of distributions. As far as "try" and "only" are concerned, they were added that late because well, nobody really had the problem (except me), and I simple couldn't/didn't think of this before. The interface was made up as I got along, which clearly shows. Show quoted text

> > Hope that clears this up, I think it might be described somewhere > > in the documentation, but it is surely buried. > > Sorry that this is so complicated.

> > That's life ;-)

If I can improve the doc or code, tell me how and I see if I can include these things into v1.89. Probably won't help you,but maybe the next guy :) best wishes, Tels PS: Using "GMP" over "Calc" is not a simple black-and-white issue, because for small numbers GMP is actually slower than Calc. So the decision on wether to use GMP or Calc or whatever should really be left to the application that loades the modules. Yes, that makes it more complicated for the app while this detail should be hidden, but this cannot be easily changed (unless someone invents a way to speed up the GMP library for very small numbers - which is unlikely :) -- Signed on Thu Apr 10 23:15:08 2008 with key 0x93B84C15. View my photo gallery: http://bloodgate.com/photos PGP key on http://bloodgate.com/tels.asc or per email. "We're looking at a future where only the very largest companies will be able to implement software, and it will technically be illegal for other people to do so." -- Bruce Perens, 2004-01-23

Download (untitled)
application/pgp-signature 481b

Message body not shown because it is not plain text.

Thu Apr 10 17:38:23 2008 solutions [...] overmeer.net - Correspondence added

Subject:	Re: [rt.cpan.org #34822] Which library is used?
Date:	Thu, 10 Apr 2008 23:37:54 +0200
To:	"nospam-abuse [...] bloodgate.com via RT" <bug-Math-BigInt [...] rt.cpan.org>
From:	Mark Overmeer <solutions [...] overmeer.net>

Show quoted text

> <URL: http://rt.cpan.org/Ticket/Display.html?id=34822 >

> > I understand the problems designing the interface, but it gives major > > problems, certainly when there is dynamic loading in play as well.

Show quoted text

> > > Note that switching libraries _after_ constructing already > > > Math::BigInt objects is a bad idea, as these objects cannot mix, > > > e.g. if $a was constructed library GMP and $b with Calc, you cannot > > > do "$a + $b" as this will blow up spectaculary :)

> > > > You may freeze the library at INIT or CHECK type.

our $fixed; INIT { $fixed++ } export { return if $fixed } Show quoted text

> 2007-01-27 v1.78 Tels > * implement "try" and "only" as replacements for "lib"

Quite a lot of Linux dists froze perl at 5.8.8... Math::BigInt 1.77 Show quoted text

> If I can improve the doc or code, tell me how and I see if I can include > these things into v1.89. Probably won't help you,but maybe the next > guy :)

Well, you could leave the discission to the main script only, so ignore what modules say, just what the end-user thinks is smart. -- Thanks for the help, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net

Thu Apr 10 17:56:54 2008 nospam-abuse [...] bloodgate.com - Correspondence added

Subject:	Re: [rt.cpan.org #34822] Which library is used?
Date:	Thu, 10 Apr 2008 23:56:10 +0200
To:	bug-Math-BigInt [...] rt.cpan.org
From:	Tels <nospam-abuse [...] bloodgate.com>

On Thursday 10 April 2008 23:38:24 Mark Overmeer via RT wrote: Show quoted text

> Queue: Math-BigInt > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=34822 > >

> > <URL: http://rt.cpan.org/Ticket/Display.html?id=34822 > > >

> > > I understand the problems designing the interface, but it gives > > > major problems, certainly when there is dynamic loading in play > > > as well. > > >

> > > > Note that switching libraries _after_ constructing already > > > > Math::BigInt objects is a bad idea, as these objects cannot > > > > mix, e.g. if $a was constructed library GMP and $b with Calc, > > > > you cannot do "$a + $b" as this will blow up spectaculary :)

> > > > > > You may freeze the library at INIT or CHECK type.

> > our $fixed; > INIT { $fixed++ } > export { return if $fixed }

Sorry, I am a bit to confused as what this should accomplish (and what problem it tries to solve). Can you explain this to me? Show quoted text

> > 2007-01-27 v1.78 Tels > > * implement "try" and "only" as replacements for "lib"

> > Quite a lot of Linux dists froze perl at 5.8.8... Math::BigInt 1.77

Yeah, I was checking corelist and comparing releases and CHANGES entries for the last half hour. That seems to be the problem, namely that 5.8.9 (with an updated Math::BigInt) is not yet released, and 5.10 isn't really widely used. I guess the only "fix" is to set a requirement for a certain Math::BigInt (probably v1.88 as people need to upgrade anyway) and have people upgrade Math::BigInt from CPAN. (As the person who slaved away for the changes between v1.77 and v1.88, I'd recommend upgrading anyway, alone for the bugfixes :) Show quoted text

> > If I can improve the doc or code, tell me how and I see if I can > > include these things into v1.89. Probably won't help you,but maybe > > the next guy :)

> > Well, you could leave the discission to the main script only, so > ignore what modules say, just what the end-user thinks is smart.

I am not sure that would technically work. How should f.i. Math::BigInt decided when to ignore a request to load a library - and when not? I guess it would also break things like "bigrat", which loads libraries, too, and can load a certain backend on user request, too: # perl -MData::Dumper -Mbigrat=l,GMP -wle 'print \ Dumper(Math::BigInt->config())' $VAR1 = { 'trap_inf' => 0, 'div_scale' => 40, 'trap_nan' => 0, 'version' => '1.88', 'round_mode' => 'even', 'lib_version' => '1.24', 'downgrade' => undef, 'lib' => 'Math::BigInt::GMP', 'class' => 'Math::BigInt', 'upgrade' => 'Math::BigFloat', 'precision' => undef, 'accuracy' => undef }; All the best, Tels -- Signed on Thu Apr 10 23:49:06 2008 with key 0x93B84C15. Get one of my photo posters: http://bloodgate.com/posters PGP key on http://bloodgate.com/tels.asc or per email. "...no one would risk manipulating votes in an election because it's against the law and carries a heavy penalty..." -- Diebold spokesman David Bear

Download (untitled)
application/pgp-signature 481b

Message body not shown because it is not plain text.

Thu Apr 10 18:11:31 2008 solutions [...] overmeer.net - Correspondence added

Subject:	Re: [rt.cpan.org #34822] Which library is used?
Date:	Fri, 11 Apr 2008 00:11:01 +0200
To:	"nospam-abuse [...] bloodgate.com via RT" <bug-Math-BigInt [...] rt.cpan.org>
From:	Mark Overmeer <solutions [...] overmeer.net>

Show quoted text

> > > > You may freeze the library at INIT or CHECK type.

> > > > our $fixed; > > INIT { $fixed++ } > > export { return if $fixed }

> > Sorry, I am a bit to confused as what this should accomplish (and what > problem it tries to solve). Can you explain this to me?

In some way, you should freeze the choice for a library on some moment. For instance at INIT time. Show quoted text

> I guess the only "fix" is to set a requirement for a certain > Math::BigInt (probably v1.88 as people need to upgrade anyway) and have > people upgrade Math::BigInt from CPAN.

The Unbuntu Perl maintainer was complaining today, and he was not the first one... apparently, it has a lot of other modules which are upgraded as well. Show quoted text

> I am not sure that would technically work. How should f.i. Math::BigInt > decided when to ignore a request to load a library - and when not?

Only accept library gestures in main::, because currently people can use external modules which contradict each other. I started the message exchange with the remark that interface design is very difficult. Sometimes unsolvable. -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net

Fri Apr 11 10:13:35 2008 nospam-abuse [...] bloodgate.com - Correspondence added

Subject:	Re: [rt.cpan.org #34822] Which library is used?
Date:	Fri, 11 Apr 2008 16:06:02 +0200
To:	bug-Math-BigInt [...] rt.cpan.org
From:	Tels <nospam-abuse [...] bloodgate.com>

On Friday 11 April 2008 00:11:32 Mark Overmeer via RT wrote: Show quoted text

> Queue: Math-BigInt > Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=34822 > >

> > > > > You may freeze the library at INIT or CHECK type.

> > > > > > our $fixed; > > > INIT { $fixed++ } > > > export { return if $fixed }

> > > > Sorry, I am a bit to confused as what this should accomplish (and > > what problem it tries to solve). Can you explain this to me?

> > In some way, you should freeze the choice for a library on some > moment. For instance at INIT time.

I am afraid this is not really possible, because while it hasn't been directly supported (e.g. with a method to do this easily), it has always possible to swap out the library at run-time, either by calling import() again, or with a quick eval. In fact, this is how Math::BigFloat replaces the Math::BigInt library when you ask it to load a different library. The side-effect that this makes math with two objects created with different library impossible is just that, a side-effect, that is not even nec. to occur. For instance, swapping Calc and FastCalc can be done without any negative side-effects. In fact, it would currently be possible to write a library that always uses GMP when dealing with it's numbers internally, but leaves a different library for the main program. It would be a bit cumbersome to write, but it should be possible. It would also be possible to fix the problem that $A and $B were created with different libraries, but it would come at a high cost (a check for this condition in every math operation). This slowdown is the reason why I haven't added it. (Addendum: I think I got an idea how this might be implemented without a slowdown, but I need to test it and it is a lot of work touching lots of code) Show quoted text

> > I guess the only "fix" is to set a requirement for a certain > > Math::BigInt (probably v1.88 as people need to upgrade anyway) and > > have people upgrade Math::BigInt from CPAN.

> > The Unbuntu Perl maintainer was complaining today, and he was not the > first one... apparently, it has a lot of other modules which are > upgraded as well.

He didn't complain to me yet ;) Can you please forward his email to me or ask him to resend it to me, so I can have a look what exactly his problem is? Show quoted text

> > I am not sure that would technically work. How should f.i. > > Math::BigInt decided when to ignore a request to load a library - > > and when not?

> > Only accept library gestures in main::, because currently people can > use external modules which contradict each other.

I don't think this would even work, as for instance bigrat doesn't load Math::BigInt from main::. And apart from this, this restriction is coming probably a few years to late. As for the external modules, I think the message should be that these external modules should leave the decision which library to load to the main program - this is, infact the reason for why "lib" now warns if the library couldn't be loaded - formerly these situations were silently and thus the library choice would also silently be ignored. The current system makes it more explicit. Plus it allows a main program to load a completely different library (for testing, debugging, or maybe because there is a faster library working in a few years). I'll try to explain this better in the FAQ. (See below) Show quoted text

> I started the message exchange with the remark that interface design > is very difficult. Sometimes unsolvable.

My first idea would be to write a detailed FAQ entry on how, why and when to load what library, possible replacing the already existing section in Math::BigInt. The document will be first here: http://bloodgate.com/wiki/index.php?title=POD:Math::BigInt::FAQ and later I will probably just include it into the distribution. All the best, Tels -- Signed on Fri Apr 11 16:04:49 2008 with key 0x93B84C15. Get one of my photo posters: http://bloodgate.com/posters PGP key on http://bloodgate.com/tels.asc or per email. "Most people, I think, don't even know what a rootkit is, so why should they care about it?" -- Thomas Hesse, President of Sony BMG's global digital business division, 2005.

Download (untitled)
application/pgp-signature 481b

Message body not shown because it is not plain text.

Fri Apr 11 16:19:21 2008 Mark [...] Overmeer.net - Correspondence added

Subject:	Re: [rt.cpan.org #34822] Which library is used?
Date:	Fri, 11 Apr 2008 22:17:39 +0200
To:	"nospam-abuse [...] bloodgate.com via RT" <bug-Math-BigInt [...] rt.cpan.org>
From:	Mark Overmeer <mark [...] overmeer.net>

Hi Tels, It's becoming a long debate. Time to freeze it in a FAQ, for everyone. I have removed lib/try references in my module, and added a short note in the 'performance' section which explains the possibility. Show quoted text

> My first idea would be to write a detailed FAQ entry on how, why and > when to load what library, possible replacing the already existing > section in Math::BigInt.

You know... people don't like to read. And people are in for a difficult decission, certainly when modules define conflicting backends. It should DWIM. Show quoted text

> The document will be first here: > http://bloodgate.com/wiki/index.php?title=POD:Math::BigInt::FAQ

Still looks quite empty. Success, and thanks for your extensive view in the guts of your modules. Let me known when you need me to proof-read. -- Regards, MarkOv ------------------------------------------------------------------------ Mark Overmeer MSc MARKOV Solutions Mark@Overmeer.net solutions@overmeer.net http://Mark.Overmeer.net http://solutions.overmeer.net

Fri Apr 11 16:21:03 2008 Mark [...] Overmeer.net - Correspondence added

Subject:	Re: [rt.cpan.org #34822] Which library is used?
Date:	Fri, 11 Apr 2008 22:20:15 +0200
To:	"nospam-abuse [...] bloodgate.com via RT" <bug-Math-BigInt [...] rt.cpan.org>
From:	Mark Overmeer <mark [...] overmeer.net>

Show quoted text

> > The Unbuntu Perl maintainer was complaining today, and he was not the > > first one... apparently, it has a lot of other modules which are > > upgraded as well.

> > He didn't complain to me yet ;) Can you please forward his email to me > or ask him to resend it to me, so I can have a look what exactly his > problem is?

This is what he tells me. MarkOv. -------------- 8< ----------------------- 8< --------- Queue: XML-Compile Ticket <URL: http://rt.cpan.org/Ticket/Display.html?id=34817 > On Thu Apr 10 05:30:11 2008, solutions@overmeer.net wrote: Show quoted text

> * jesper@krogh.cc via RT (bug-XML-Compile@rt.cpan.org) [080410 09:05]:

> > > > Thu Apr 10 05:04:57 2008: Request 34817 was acted upon. > > Transaction: Ticket created by jesper@krogh.cc > > Queue: XML-Compile > > Subject: Reduce dependencies.. > > Broken in: 0.73 > > > > XML::Compile has pretty harsh requirements.. on an Ubuntu Gutsy (latest > > Ubuntu release) an installation of XML::Compile 0.73 would force an > > update of the perl-installations due to the Math::BigInt requirement. > > Isn't it possible to circumvent that?

> > What do you mean with "the perl-installations" ? What needs to be > updated? The newest version from Math::BigInt is available as dual > lived module from CPAN. > > Math::BigInt is rather crucial for a correct functioning of XML::Compile > (because the XML Schema spec defines that a "Decimal" should support at > least 18 digits)

Sorry if I'm too distribution specific. Ubuntu Gutsy provides Math::BigInt 1.77, which is delivered by the perl-modules, build by the perl source package that also provices the perl-interpreter. So in order to create an updated package of this with the new Math::BigInt to push out to our servers I have to go through and build and update a new'er version of "perl-modules" and "perl". So my question was.. is 1.78 of Math::BigInt really a hard requirement? since it is giving quite lagre update challenges. Thanks.

Mon Apr 21 15:26:47 2008 TELS [...] cpan.org - Taken

Mon Apr 21 15:28:34 2008 TELS [...] cpan.org - Correspondence added 15 min

With the amended documentation in v1.89, this issue should be resolved. If it isn't or if you have comments, please do not hesitate to reopen this bug report by replying to this email :) Thank you for your report!

Mon Apr 21 15:28:38 2008 TELS [...] cpan.org - Status changed from 'open' to 'resolved'

Mon Apr 21 15:28:59 2008 TELS [...] cpan.org - Broken in 1.88 added