Bug #88932 for PDL-Stats: won't support 64bit indexing in PDL-2.007

Tue Sep 24 09:29:39 2013 chm [...] cpan.org - Ticket created

Subject:

won't support 64bit indexing in PDL-2.007

Hi Maggie-

I just tested the latest developers release and PDL::Stats built and tested
fine on my win7/cygwin PDL install (test report is submitted). Now that
that is out of the way, I looked through some of the PP code and noticed
some usages that will definitely cause problems if someone uses a
64bit sized piddle.

See

http://mailman.jach.hawaii.edu/pipermail//pdl-porters/2013-September/005900.html

for a check list of things to look for and fix. The key is to avoid using int sized
variables for values that loop over index dimensions, offsets, of number of
elements or bytes. If you search the .pp files for long or int you'll find places
that may need a fix. E.g., in glm.pp lines:

151     long i, j;
164         j = (long) ((i-1) * (double)(rand()) / (double)(RAND_MAX) + .5);
216     long N = $SIZE(n);
228     long N = 0;
265     long N = $SIZE(n);
279     long N = 0;
363     long N = 0;
394     long N = $SIZE(n);
403     long N = 0;
480     long N = $SIZE(n);
493     long N = 0;

You get the idea. Still to come on the PDL main distribution side is support
for back-compatibility and portability (probably by adding something to PP
creating a compatibility type that works with older PDL versions). In the
meantime, you could also just put a runtime check for defined(&PDL::indx)
and issue a warning that 64bit support is not there yet....

Tue Sep 24 10:11:13 2013 MAGGIEXYZ [...] cpan.org - Correspondence added

Thanks for checking, Chris! I'll have some time to look into it over the weekend and hopefully have it ready for 64bit sized piddle soon. Best, Maggie On 2013-09-24 09:29:39, CHM wrote: Show quoted text

> Hi Maggie- > > I just tested the latest developers release and PDL::Stats built and > tested > fine on my win7/cygwin PDL install (test report is submitted). Now > that > that is out of the way, I looked through some of the PP code and > noticed > some usages that will definitely cause problems if someone uses a > 64bit sized piddle. > > See > > http://mailman.jach.hawaii.edu/pipermail//pdl-porters/2013- > September/005900.html > > for a check list of things to look for and fix. The key is to avoid > using int > sized > variables for values that loop over index dimensions, offsets, of > number of > elements or bytes. If you search the .pp files for long or int you'll > find > places > that may need a fix. E.g., in glm.pp lines: > > 151 long i, j; > 164 j = (long) ((i-1) * (double)(rand()) / (double)(RAND_MAX) + .5); > 216 long N = $SIZE(n); > 228 long N = 0; > 265 long N = $SIZE(n); > 279 long N = 0; > 363 long N = 0; > 394 long N = $SIZE(n); > 403 long N = 0; > 480 long N = $SIZE(n); > 493 long N = 0; > > You get the idea. Still to come on the PDL main distribution side is > support > for back-compatibility and portability (probably by adding something > to PP > creating a compatibility type that works with older PDL versions). In > the > meantime, you could also just put a runtime check for > defined(&PDL::indx) > and issue a warning that 64bit support is not there yet....

Tue Sep 24 10:11:13 2013 The RT System itself - Status changed from 'new' to 'open'

Tue Sep 24 10:11:13 2013 MAGGIEXYZ [...] cpan.org - Taken

Mon May 30 15:42:58 2016 ETJ [...] cpan.org - Correspondence added

Hi Chris, I believe this issue is now resolved and can be closed. Do you agree?

Mon May 30 20:58:18 2016 devel.chm.01 [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #88932] won't support 64bit indexing in PDL-2.007
Date:	Mon, 30 May 2016 20:58:07 -0400
To:	bug-PDL-Stats [...] rt.cpan.org
From:	Chris Marshall <devel.chm.01 [...] gmail.com>

I haven't looked at the new PDL::Stats release. If the changes I noted earlier are fixed it is probably ok. Tests with large piddles are tricky since it is easy to get into swap hang unless you have a system with very large memory. I have access to one, but I don't know how long I will take me to check things out. I'll try for a spot check tomorrow. --Chris On 5/30/2016 15:43, Ed J via RT wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=88932 > > > Hi Chris, > > I believe this issue is now resolved and can be closed. Do you agree?

Tue May 31 15:18:59 2016 devel.chm.01 [...] gmail.com - Correspondence added

Subject:	Re: [rt.cpan.org #88932] won't support 64bit indexing in PDL-2.007
Date:	Tue, 31 May 2016 15:18:48 -0400
To:	bug-PDL-Stats [...] rt.cpan.org
From:	Chris Marshall <devel.chm.01 [...] gmail.com>

Looks ok by inspection to me. I don't have any problems or a system with enough memory to check if the algorithms work. Maybe Maggie can cook up something sort of analytic to test with. Otherwise, I would say close the ticket with a note to the effect that the port to 64bit indexing has taken place but has not been fully exercised. --Chris On Mon, May 30, 2016 at 8:58 PM, devel.chm.01@gmail.com via RT < bug-PDL-Stats@rt.cpan.org> wrote: Show quoted text

> <URL: https://rt.cpan.org/Ticket/Display.html?id=88932 > > > I haven't looked at the new PDL::Stats release. > If the changes I noted earlier are fixed it is > probably ok. Tests with large piddles are tricky > since it is easy to get into swap hang unless > you have a system with very large memory. I have > access to one, but I don't know how long I will > take me to check things out. I'll try for a spot > check tomorrow. > > --Chris > > On 5/30/2016 15:43, Ed J via RT wrote:

> > <URL: https://rt.cpan.org/Ticket/Display.html?id=88932 > > > > > Hi Chris, > > > > I believe this issue is now resolved and can be closed. Do you agree?

> > >

Tue May 31 19:51:26 2016 ETJ [...] cpan.org - Correspondence added

On Tue May 31 15:18:59 2016, devel.chm.01@gmail.com wrote: Show quoted text

> Looks ok by inspection to me. I don't have any problems or a system with > enough memory to check if the algorithms work. Maybe Maggie can cook up > something sort of analytic to test with. Otherwise, I would say close the > ticket with a note to the effect that the port to 64bit indexing has taken > place but has not been fully exercised.

I will do that. Please feel free to reopen this ticket or to open a new one if you have further issues!

Tue May 31 19:51:36 2016 ETJ [...] cpan.org - Status changed from 'open' to 'resolved'

Tue May 31 21:41:33 2016 maggiexyz [...] users.sourceforge.net - Correspondence added

Subject:	Re: [rt.cpan.org #88932] won't support 64bit indexing in PDL-2.007
Date:	Tue, 31 May 2016 21:41:04 -0400
To:	bug-PDL-Stats [...] rt.cpan.org
From:	Maggie Xiong <maggiexyz [...] users.sourceforge.net>

Shall I make some unit tests that exercise huge pdls? :P Srsly thou, I'll try to get a beefy box to test it. Feel free to ping me if I haven't get back about it in a couple weeks. On Tue, May 31, 2016 at 7:51 PM, Ed J via RT <bug-PDL-Stats@rt.cpan.org> wrote: Show quoted text

> Queue: PDL-Stats > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=88932 > > > On Tue May 31 15:18:59 2016, devel.chm.01@gmail.com wrote:

> > Looks ok by inspection to me. I don't have any problems or a system with > > enough memory to check if the algorithms work. Maybe Maggie can cook up > > something sort of analytic to test with. Otherwise, I would say close

> the

> > ticket with a note to the effect that the port to 64bit indexing has

> taken

> > place but has not been fully exercised.

> > I will do that. Please feel free to reopen this ticket or to open a new > one if you have further issues! >