Should we add Lazy Loading?

[Timu57]

Currently we save and return all found documents in an array.

A solution approach for lazy loading could be that we create an object called Results or something that extends Iterable.

So the user can use it like an array.

The Results object stores an array containing just the id's of all files found.

When the user now iterates of Results the document will be loaded and returned. (file will be called with it's _id)

To make functionality like Group By possible we could store all found documents in a new location as files and load them from there.

That is a pretty bare bone solution and could possibly result in slower speeds but could help keeping the memory usage low.

Do you have an idea for a better solution?
Do you think we don't need lazy loading?

I really want to hear your thoughts and opinions.

[rakibtg]

@Timu57 Nice idea! When do we clean those temporary files?
However, I dont think we need lazy loading feature at this moment due to the caching layer that we have!

[Timu57]

@rakibtg Ok, if you mean we don't need it currently :)

That came into my mind because we save all results into a single array and if the database is big the array could take too much ram..

[rakibtg]

@Timu57 Correct! Later we can stream a file instead of loading all the contents in memory to tackle this problem.