Online generation of large realizations

[juliohm]

We are working on projects where each realization has more than 200 million geometries. Assuming that values are stored as Float64, we have ~0.5Gb per realization. If we request 10 parallel processes, we can consume up to 5Gb of RAM.

Given these RAM constraints, it would be useful to introduce an OnlineEnsemble or some other mechanism to consume realizations in an online fashion. The pseudo-code for this could look something like the following:

# request N realizations with P processes
reals = rand(proc, domain, N, pool = 1:P, async = true)

while !isempty(reals)
  real = pop!(reals)
  # do work
  # save to disk
end

In this case the rand call would return immediately, and the N realizations would be consumed on batches by P parallel processes. At any single point in time, the maximum RAM required will be a function of P as opposed to N.