Suggestion : "Lazy iterators" like "()" in addition to "[]" as in Python

[serge-hulne]

Proposal

• Create a second syntax to handle data coming from streams (files, web) as lazy operators:

• () vs [] as in python

• Example:

A = [f(x) for x in V]

A is stored as an array (the entire collection of values [f(0), f(1), ... f(N)] ) are stored in RAM.

However if one uses:

A = (f(x) for x in V)

Background

In Python if one uses the structure:

A = [f(x) for x in V]

A is stored as an array (the entire collection of values [f(0), f(1), ... f(N)] ) are stored in RAM.

However if one uses:

A = (f(x) for x in V)

Then A is a mere iterator and the values will only be computed as needed.

• With the array notation "[]", in a cascade of list comprehensions, all the intermediary results will therefore be kept in RAM.

• If one works with a very large structure and does a large number of successive transformations on it (typically in the field of NLP (see FLTK, for instance)), then the set of intermediary results can cause excessive usage of memory.

• Exemple of code where a "lazy operator" (implemented as a channel) gets transformed into an array by the current goplus handling of list comprehension

import "fmt"

type Counter struct {
	Value int
	HasNext bool
}

func (c *Counter) Next() {
	if c.Value <10 {	
		c.Value++
		c.HasNext = true
	} else {
		c.HasNext = false
	}
}

func GeneratortoIterator(c Counter) chan int {
	ch := make(chan int)
	go func() {
		defer close(ch)		
		for c.HasNext {
			ch <- c.Value
			c.Next()
		}
	}()
	return ch
}


func main() {

	c := Counter{0, true} 

	it := GeneratortoIterator(c) 
	// it is a channel (lazy iteraor)

	v1 := [x for x <- it]
	// v1 is an array at this point

	fmt.Printf("Type: v1 %T\n", v1)

	for k, v <- v1 {
		println "v[", k, "] = ", v
	}

	v2 := [x * 2 for x <- v1]

	println "v2  = ", v2

}

Workarounds

One way to code it consists in substituting arrays with channels, as in my example, above.

[serge-hulne]

Here is one possible implementation in Go for lazy iterators and associated functions (filter, map, reduce, etc.):

https://github.com/serge-hulne/go_iter

PS: I refactored it to make it easier to use.