Skip Menu |

This queue is for tickets about the Math-NumSeq CPAN distribution.

Report information
The Basics
Id: 83578
Status: resolved
Priority: 0/
Queue: Math-NumSeq

People
Owner: Nobody in particular
Requestors: DANAJ [...] cpan.org
Cc:
AdminCc:

Bug Information
Severity: Normal
Broken in: 56
Fixed in: (no value)



Subject: Iterated Totient uses too much memory
time perl -MMath::NumSeq::Totient -E 'my $seq = Math::NumSeq::Totient->new; foreach my $i (1 .. 10_000_000) { my($i,$value) = $seq->next; }' 39.3s seconds and 1.4 GB of memory. time perl -MMath::NumSeq::Totient -E 'my $seq = Math::NumSeq::Totient->new; foreach my $i (1 .. 10_000_000) { $seq->ith($i); }' 28.1s and basically no memory. It seems more efficient in time and *much* more efficient in space to just calculate the totient each time. Or come up with an alternate sieve. I get the same result when looking at smaller ranges such as 100k. For comparison with Math::Prime::Util 0.21: time perl -Iblib/lib -Iblib/arch -MMath::Prime::Util=:all -E 'foreach my $i (1 .. 10_000_000) { euler_phi($i); }' 27.6s (it's 19.8s if I remove the expensive input validation). Basically no memory since it is just factoring. time perl -Iblib/lib -Iblib/arch -MMath::Prime::Util=:all -E 'euler_phi(1,10_000_000);' 0.8s (returns an array of all the totients using a sieve). Can be segmented to use a bit less memory.
Since TotientCumulative uses the sieve from Totient, it also has the memory and speed issues, whether iterated or not. This hack makes TotientCumulative run slightly faster with next, and not blow up memory: At top: *_ith_totient = \&Math::NumSeq::Totient::ith; In rewind: $self->{'sum'} = -1; In next: return ($i, $self->{'sum'} += _ith_totient($self,$i)); The sum change is a hack because Totient->ith(0) returns 1 instead of 0 like _totient_by_sieve(0). Technically it's undefined, but I think SAGE has the right idea: "We defined this to be the number of positive integers <= n that are relatively prime to n. Thus if n<=0 then euler_phi(n) is defined and equals 0." I find that helpful for other functions. Timings for TotientCumulative with ... foreach my $i (1..10_000_000) { my (undef,$value) = $seq->next; }: 34.4s Math::Prime::Util::euler_phi($i) 35.5s _ith_totient($self,$i)); 40.7s _totient_by_sieve($self,$i)) ** 1.4GB of memory ** The next version of MPU will have a totient directly in XS which makes the time 12-15s. ith($i) could be done using List::Util::sum(euler_phi(0,$i)) which takes slightly over 1s for 10M.
Thanks. I think the sieve was first and seemed like a good idea at the time. I didn't realize the prime factorization was now better. Show quoted text
> Since TotientCumulative uses the sieve from Totient,
Yes, about four places share the sieve. Show quoted text
> The sum change is a hack because Totient->ith(0) returns 1 instead of 0 > like _totient_by_sieve(0). Technically it's undefined,
Hmm. Perhaps it was never meant for i=0. I probably mean i<=0 in the "undef" condition of the ith() func of Totient.pm for a start. Show quoted text
> ith($i) could be done using > List::Util::sum(euler_phi(0,$i)) which takes slightly over 1s for 10M.
That pushes $i many return values on the stack does it (the perl stack that is)? Something done in chunks might be a compromise between size and time. I had meant to try calling some of your funcs as available, but never got around to it. In a few places I had the urge to try to support bignums which are entirely small prime factors with some sort of limit on the trial division based on the size of the input. Doing so can sometimes get answers when plotting big or slightly big numbers. Though where to put a limit is a bit arbitrary.
On Sat Feb 23 19:20:20 2013, KRYDE wrote: Show quoted text
> Thanks. I think the sieve was first and seemed like a good idea at the > time. I didn't realize the prime factorization was now better.
It's also possible that different machines or arguments would result in different timings. I just did timing on a couple x86 Linux machines. Show quoted text
> > The sum change is a hack because Totient->ith(0) returns 1 instead of 0 > > like _totient_by_sieve(0). Technically it's undefined,
> > Hmm. Perhaps it was never meant for i=0. I probably mean i<=0 in the > "undef" condition of the ith() func of Totient.pm for a start.
Quite possible. Apostol and Koblitz define it only for n >= 1. Hardy and Wright define it as "the number of positive integers not greater than and prime to m, that is to say the number of integers n such that 0 < n <= m, (n,m) = 1" which would support the idea that phi(n) = 0 for n <= 0. Wolfram says: "By convention, phi(0)=1, although Mathematica defines EulerPhi[0] equal to 0 for consistency with its FactorInteger[0] command." Show quoted text
> > ith($i) could be done using > > List::Util::sum(euler_phi(0,$i)) which takes slightly over 1s for 10M.
> > That pushes $i many return values on the stack does it (the perl stack > that is)? Something done in chunks might be a compromise between size > and time.
Yes, doing the one big sum could lead to big memory issues. Segmenting it would be better. Right now MPU's ranged moebius function is segmented, but the totient isn't. It can still be helpful since the C array is a lot smaller, but at some point I'll need to do it right. I was interested in ranged totients for things like Project Euler solutions, but I'd never come across a need for the summation. I added the Mertens function separately for summing the Möbius function, as there are efficient ways to calculate it. Show quoted text
> I had meant to try calling some of your funcs as available, but never > got around to it. In a few places I had the urge to try to support > bignums which are entirely small prime factors with some sort of limit > on the trial division based on the size of the input. Doing so can > sometimes get answers when plotting big or slightly big numbers. Though > where to put a limit is a bit arbitrary.
No problem -- there is something to be said for using well tested modules. The one thing it would allow is 64-bit (or bignum) support for a lot of functions. The fast prime_count and nth_prime might be nice. divisor_sum and jordan totient help with a few more OEIS series. I've thought about making a list of OEIS sequences and the code to generate them, especially for the ones where they're ~ one-liners. As for the bignum idea, do you mean doing a gcd on primorials? I have some areas that do GCDs on primes up to 20046, but they're just for speed of quick primality checking on big (512 - 4096 bit) numbers. It's very fast with GMP or Pari, but the default BigInt backend is deathly slow for big GCDs. It looks like for factoring on my 64-bit machine, MPU averages under 100 microseconds for factoring random 19-digit numbers. Restricting to numbers with no factors under 100 still results in times under 1 millisecond per number. In fact, it's faster than calling Math::Pari::factorint for almost all 64-bit inputs. My SQUFOF code could still use a little more optimization. Pari is definitely better once we get past 25 or so digits, as my ECM implementation is pretty basic and I don't have a QS implemented. Still, it should let you factor most 30 digit numbers in under a second (that's pathetic by the standards of yafu, msieve, etc. but pretty good for Perl).
On Sat Feb 23 21:44:07 2013, DANAJ wrote: Show quoted text
> I've > thought about making a list of OEIS sequences and the code to generate > them, especially for the ones where they're ~ one-liners.
I've tried to catalogue the A-numbers of what I wrote, but mainly as a cross reference rather than any attempt at coverage. The easy ones tend not to be very interesting, and the hard ones take work :-). (Math::NumSeq::OEIS makes a sequence by A-number, and $seq->oeis_anum() is the A-number of a sequence if it has one and if I knew it to put it in.) Show quoted text
> As for the bignum idea, do you mean doing a gcd on primorials?
Oh, no I just meant that say N=2^50 * 3^100 is a bignum but it's entirely small prime factors so tractable for various calculations involving the factorization. Some of the "path" X,Y -> N things I've tinkered with sometimes make such values.
On Sun Mar 03 19:12:32 2013, KRYDE wrote: Show quoted text
> Oh, no I just meant that say N=2^50 * 3^100 is a bignum but it's > entirely small prime factors so tractable for various calculations > involving the factorization. Some of the "path" X,Y -> N things I've > tinkered with sometimes make such values.
I see. I use something vaguely like that for the BLS primality proving in Math::Prime::Util::GMP, where I can do the proof if I can get factors up to (n/2)^1/3 of n-1. I put in a quick option that does trial division to 1000, some rounds of Pollard-Rho, then exits if it hasn't found enough factors. For regular is_prime on numbers less than 200 bits I have it run the quick option. It only takes 2ms or so to prove 70% of 100-bit numbers (admittedly I don't need to factor them all completely). It might be an interesting option to add something like that with an external interface. It could be done with alarms, but (1) not all systems support them, and (2) we'd like to get a partial factorization back. So just something like factor_partial($n, $effort) where effort is either a scale 1-3 or a time in ms (which would be really approximate). It would return a list of factors which may not all be prime. It'd be nice to have some way of indicating if it was a partial factorization -- it could return ([p,p,p],[c,c,c]) instead I suppose (p being a prime factor, c being a composite factor).
I think this can be closed. It could always be optimized more, but the excessive memory use is gone. Times on a different machine than in the original message. 59.8s 1447MB Math::NumSeq 56 $seq->next 44.6s 5MB Math::NumSeq 56 $seq->ith 70.5s 5MB Math::NumSeq 67 $seq->next 41.4s 5MB Math::NumSeq 67 $seq->ith 10.5s 10MB MPU 0.36 individual calls 5.3s 21MB MPU 0.36 iterator using 100k ranges I've been trying to reduce the size of MPU. Looks like 2MB of that is loading Math::BigInt, 3MB is the main PP code (only used when XS and GMP can't do it), and 0.5MB is random primes. It's under 5MB with those removed.
One last comment relating to the previous one... https://metacpan.org/release/Math-Prime-Util 0.37 now uses less memory than using Math::NumSeq::PrimeFactorCount. It also doesn't have any problems with numbers larger than 2^32 (Math::NumSeq::Totient(4294967296) returns undef because Math::Factor::XS uses trial division).
On Sun Jan 19 02:13:50 2014, DANAJ wrote: Show quoted text
> I think this can be closed.
Beaut, done.