Bug #102673 for List-MoreUtils: Improving Iterator / Transformation functions ...

Tue Mar 10 10:25:06 2015 REHSACK [...] cpan.org - Ticket created

CC:	"Michael G. Schwern" <schwern [...] pobox.com>, "Toby Inkster" <mail [...] tobyinkster.co.uk>, ilmari+cpan [...] ilmari.org, blue [...] thisisnotmyrealemail.com, smls75 [...] gmail.com, daviddlowe.flimm [...] gmail.com, "David Golden" <xdg [...] xdg.me>, ether [...] cpan.org

Hi, I'm currently stepping through the wishlist of LMU to see what's next. All of you (beside xdg@ - but he has some overall perspective meanwhile) are reasonable part of a common wish for improved transformation / iteration functions: Beside the concrete use-case functions (array_eq, ltsorted, array_map, ...) - I'd like to summon the requirements to of the central loops: 1) loops should be done with BLOCK (callback) and iterator blockwise N, CODE, LIST windowwise N, CODE, LIST # (4 * 7 ^3, 5 * 3 ^ 2, 6 * 19 ^ 1) # reduce can sum it up ... blockwise 3 { $_[0] * $_[1] ** $_[2] } (4, 7, 3, 5, 3, 2, 6, 19, 1); # (1,1,1,0,1,1,0) # could be used with all for ltsorted windowwise 2 { $_[0] < $_[1] } (1,2,3,5,4,6,8,7); How should the iterator functions be named? blockwise_i? chunks? windowwise_i? magnifier? 2) the count of items per turn should be variable (always? sometimes?) This affects pairwise only at the moment - and there it's fine. But how can we (reasonable) add functions which does tuplewise as tuplewise BLOCK, (LIST) x N groupedtuplewise N, BLOCK, (LIST) x M as groupedtuplewise 2 { my ($g1, $g2, $g3, $g4) = @_; ($g1->[0] + $g2->[0]...) * (...$g4->[1]) } @L1, @L2, @L3, @L4 Do we want a groupedtuplewise_iter as my $it = groupedtuplewise_iter 2 @L1, @L2, @L3, @L4; while( my ($g1, $g2, $g3, $g4) = $it->() ) { say ($g1->[0] + $g2->[0]...) * (...$g4->[1]); } 3) mixin ... while tuplewise could be a reasonable step between pairwise and chunks from multiple array at ones, how does the windowing / magnifying fit's that picture. As a reasonable consequence, there should be a function which allows accessing overlapping chunks from multiple list. Also - currently the overlapping loop has a step of one - should this increment limited to 1, or is another increment sane? Do we want dedicated names for _step1 (eg. tuplewise vs. groupedtuplewise - or whatever ^^), or do we introduce prefixes and suffixes for that (Spreeing Helm of the Vampire)? Cheers Jens

Wed Mar 11 15:56:51 2015 REHSACK [...] cpan.org - Subject changed from (no value) to 'Improving Iterator / Transformation functions ...'

Wed Mar 11 17:15:15 2015 dagolden [...] cpan.org - Correspondence added

For iterator names, there are already several with "each_*" as a pattern. I'd find a way to continue that. Even if it's slightly odd, the naming consistency will lower the learning curve. I find "blockwise" potentially confusing as a term, since we talk about BLOCK as a code reference. Too bad we already have "natatime" doing something else. I also find "windowwise" an odd phrase. It also strikes me that both blockwise and windowwise are special cases of a function that examines N at a time and advances the starting point by M each time. I wonder if having a general purpose function is better. E.g. # look at 3 at a time, advancing by 1 each time @res = chunked( 3, 1, sub { ... }, @list ); The iterator generator could be "each_chunk". How about just "tuples" for the bringing together the elements of several lists: @t = tuples { [@_] } \@list1, \@list2, ... \@listN; Then the iterator could come from "each_tuple". Possibly relate it to "zip" like "zip_tuples" or something. I can't even understand what groupedtuplewise is supposed to do (splice input lists and feed array of arrays to function?), so I need a better example. My snap reaction is that I don't know when I'd use it, and that if I did use it, it would hinder code clarity rather than enhance it. I vote YAGNI.

Wed Mar 11 17:15:16 2015 The RT System itself - Status changed from 'new' to 'open'

Thu Mar 12 01:37:01 2015 REHSACK [...] cpan.org - Correspondence added

Without going deeper in everything just for groupedtuplewise: It was proposed as a combination of chunked and tuples. For each of following arrays, call me with chunk of N iterated by step of O. I like the idea with each_* and likely zip_* :) This bikeshedding has the goal of keeping low learning curve - so your proposals are best \o/

Thu Mar 12 03:38:27 2015 zmughal [...] cpan.org - Cc ZMUGHAL added

Fri Mar 13 13:58:57 2015 dagolden [...] cpan.org - Correspondence added

Another term that occurs to me is "pivot". Given [a, b, c], [d, e, f], [g, h, i], [j,k,l] the pivot of that would be [a, d, g, j], [b, e, h, k], [c, f, i, l]. I like that even better than "tuples" or "zip_tuples". And then the iterator would be "each_pivot".

Fri Mar 13 15:45:32 2015 dagolden [...] cpan.org - Correspondence added

The other general approach that might make sense is to allow an iterator function in place of an array reference. Then it would be fairly trivial to combine primitives. Also, consider having "chunked" and "each_chunk" return array *references* rather than lists, which I think helps with combinations like this. For instance, given @a = qw/a b c d/ and @b = qw/e f g h/ and @c = qw/i j k l/, consider what this would do: @out = pivot( each_chunk(2,2,\@a), each_chunk(2,2,\@b), each_chunk(2,2,\@c), ) I think that would give (omitting qw//): ( [ [a b], [e f], [i j] ], [ [c d], [g h], [k l] ], ) Which I think is similar to what the "grouped-tuple-wise" thing was doing. Thus I think we get a lot of generalized list manipulation just by implementing the following: * "chunked" to divide a list into array references (possibly with overlap) * "pivot" to pivot an AOA * iterator versions for those two (each_chunk and each_pivot) * things that take array references also allowing iterators instead I think the other thing necessary would be "flatten" which would take a list of array references and flatten them into a list. So "flatten" on the @output above: @flat = flatten @output Giving: ( [a b], [e f], [i j], [c d], [g h], [k l] ) Or flatten that again: @flatter = flatten @flat Giving: ( a, b, e, f, i, j, c, d, g, h, k, l ) Figuring out when to do lists and when to do array references probably takes more thought that I put in right now, but done right, I think very complicated transformations can be built up from simple primatives.

Tue Mar 17 08:31:17 2015 REHSACK [...] cpan.org - Correspondence added

On Fri Mar 13 15:45:32 2015, DAGOLDEN wrote: Show quoted text

> The other general approach that might make sense is to allow an > iterator function in place of an array reference. Then it would be > fairly trivial to combine primitives.

I have to dig a little how to implement that in suitable manner, but it sounds very reasonable. Also - I have no clue how this affects the typical prototype declaration, LMU consequently uses... I have to play a bit with that ... Show quoted text

> Also, consider having "chunked" and "each_chunk" return array > *references* rather than lists, which I think helps with combinations > like this.

How about chunk, each_chunk, chunks, each_chunks... Show quoted text

> For instance, given @a = qw/a b c d/ and @b = qw/e f g h/ and @c = > qw/i j k l/, consider what this would do: > > @out = pivot( > each_chunk(2,2,\@a), > each_chunk(2,2,\@b), > each_chunk(2,2,\@c), > )

Which could then be written as @out = pivot( chunks(2,2,\@a,\@b,\@c) ); or @out = pivot( each_chunks(2,2,\@a,\@b,\@c) ); for iterator version. Pivot could be distinguished to tuples by pivot always delivers AOA grouped by input rows and tuples takes a CODE which allows own implementation. Show quoted text

> I think that would give (omitting qw//): > > ( > [ [a b], [e f], [i j] ], > [ [c d], [g h], [k l] ], > ) > > Which I think is similar to what the "grouped-tuple-wise" thing was > doing.

Yes, that's true. So above showed "pivot" can be done by sub pivot { tuples { [@_] } @_ } Show quoted text

> Thus I think we get a lot of generalized list manipulation just by > implementing the following: > > * "chunked" to divide a list into array references (possibly with > overlap) > * "pivot" to pivot an AOA > * iterator versions for those two (each_chunk and each_pivot) > * things that take array references also allowing iterators instead

That consequently implies a larger rework of the existing functions to have a similar kind of usage everywhere. Additionally I'd like to see tuples (pivot like but have CODE instead of implicit pivot) and similar for chunk(ed). OTOH we can claim that grep, map or List::Util's reduce can do the job on the returned lists. (KISS) Show quoted text

> I think the other thing necessary would be "flatten" which would take > a list of array references and flatten them into a list. > > So "flatten" on the @output above: > > @flat = flatten @output > > Giving: > > ( [a b], [e f], [i j], [c d], [g h], [k l] ) > > Or flatten that again: > > @flatter = flatten @flat > > Giving: > > ( a, b, e, f, i, j, c, d, g, h, k, l ) > > Figuring out when to do lists and when to do array references probably > takes more thought that I put in right now, but done right, I think > very complicated transformations can be built up from simple > primatives.

That's mentioned in https://rt.cpan.org/Ticket/Display.html?id=17230 - but I like flatten more than arrify :)

Tue Mar 17 10:08:37 2015 xdg [...] xdg.me - Correspondence added

CC:	blue [...] thisisnotmyrealemail.com, daviddlowe.flimm [...] gmail.com, Karen Etheridge <ether [...] cpan.org>, ilmari+cpan [...] ilmari.org, Toby Inkster <mail [...] tobyinkster.co.uk>, Michael G Schwern <schwern [...] pobox.com>, smls75 [...] gmail.com, zmughal [...] cpan.org
Subject:	Re: [rt.cpan.org #102673] Improving Iterator / Transformation functions ...
Date:	Tue, 17 Mar 2015 10:07:55 -0400
To:	bug-List-MoreUtils [...] rt.cpan.org
From:	David Golden <xdg [...] xdg.me>

On Tue, Mar 17, 2015 at 8:31 AM, Jens Rehsack via RT < bug-List-MoreUtils@rt.cpan.org> wrote: Show quoted text

> Additionally I'd like to see tuples (pivot like but have CODE instead of > implicit pivot) and similar for chunk(ed). OTOH we can claim that grep, map > or List::Util's reduce can do the job on the returned lists. (KISS) > >

I was thinking KISS and the Unix principle of combining simple, single-purpose tools for more complex behavior. David

Tue Mar 17 10:12:40 2015 REHSACK [...] cpan.org - Correspondence added

On Tue Mar 17 10:08:37 2015, xdg@xdg.me wrote: Show quoted text

> [...] > I was thinking KISS and the Unix principle of combining simple, > single-purpose tools for more complex behavior.

Well - even if also speed is a general approach of LMU I think, let's start with KISS and see where this leads us. That's anyway very enough for a distinct release ;)

Mon May 18 05:47:16 2015 ribasushi [...] leporine.io - Cc RIBASUSHI added

Thu Aug 13 14:23:49 2015 ether [...] cpan.org - Reference by ticket #106399 added