Enable meta parallel Circuitscape - call Circuitscape on several INI files at once with 1 process each

[ranjanan]

Should fix #236

[ranjanan]

@VLucet and @vlandau could you check if this PR works for a list of INI files?

[VLucet]

Hi @ranjanan, thanks for enabling this. One question that comes to mind: could julia recognize that the input is an array when calling compute and dispatch to compute_metaparallel, and would that be a good design idea or not?

In all cases, I gave it a try, and first, I run into the same problem than I mentioned to @ViralBShah in #236 , when Circuitscape is only installed in the project environment (and not in the default environment), I get the following error, presumably because the workers inherit the default environment (which does not see Circuitscape).

julia> compute_metaparallel(ini_list, 5)
[ Info: 2020-06-17 21:37:32 : Starting up 5 processes
ERROR: On worker 2:
ArgumentError: Package Circuitscape not found in current path:
- Run `import Pkg; Pkg.add("Circuitscape")` to install the Circuitscape package.

require at ./loading.jl:892
eval at ./boot.jl:331
top-level scope at none:0
eval at ./boot.jl:331
#101 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:290
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:79
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:88
#94 at ./task.jl:358

...and 4 more exception(s).

Stacktrace:
 [1] sync_end(::Array{Any,1}) at ./task.jl:316
 [2] macro expansion at ./task.jl:335 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/macros.jl:217
 [4] macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/macros.jl:201 [inlined]
 [5] myaddprocs(::Int64) at /opt/julia/packages/Circuitscape/z0iGj/src/utils.jl:146
 [6] compute_metaparallel(::Array{String,1}, ::Int64) at /opt/julia/packages/Circuitscape/z0iGj/src/utils.jl:110
 [7] top-level scope at REPL[8]:1

Once the same version of circuitscape is installed (I made sure to install fro the same sha on the last commit on RA/metapar) in the default environment, it works fine :

julia> compute_metaparallel(ini_list, 5)
[ Info: 2020-06-17 21:46:20 : Starting up 5 processes
[ Info: 2020-06-17 21:46:32 : Launching 29 Circuitscape jobs among 5 processes
[ Info: 2020-06-17 21:48:08 : Precision used: Double
[ Info: 2020-06-17 21:48:08 : Precision used: Double
[ Info: 2020-06-17 21:48:10 : Precision used: Double
[ Info: 2020-06-17 21:48:10 : Precision used: Double
[ Info: 2020-06-17 21:48:10 : Precision used: Double
[ Info: 2020-06-17 21:48:16 : Reading maps
[ Info: 2020-06-17 21:48:17 : Reading maps
[ Info: 2020-06-17 21:48:18 : Reading maps
[ Info: 2020-06-17 21:48:19 : Reading maps
[ Info: 2020-06-17 21:48:19 : Reading maps
[ Info: 2020-06-17 21:48:21 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:22 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:23 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:24 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:24 : Resistance/Conductance map has 6245866 nodes
[ Info: 2020-06-17 21:48:34 : Total number of pair solves = 1
[ Info: 2020-06-17 21:48:34 : Solving pair 1 of 1
[ Info: 2020-06-17 21:49:01 : Solver used: AMG accelerated by CG
[ Info: 2020-06-17 21:49:01 : Graph has 6239984 nodes, 2 focal points and 1 connected components
[ Info: 2020-06-17 21:49:16 : Time taken to construct preconditioner = 13.576540371 seconds
[...]

However I then downgraded Circuitscape to the release version in the default environment and it appears that when the versions in the project and in the default environment do not coincide, we have errors. I know that sounds like I might have done something wrong.

julia> compute_metaparallel(ini_list, 5)
[ Info: 2020-06-17 22:12:06 : Starting up 5 processes
[ Info: 2020-06-17 22:12:17 : Launching 29 Circuitscape jobs among 5 processes
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Reading maps
[ Info: 2020-06-17 22:12:18 : Precision used: Double
[ Info: 2020-06-17 22:12:18 : Reading maps
[ Info: 2020-06-17 22:12:18 : Reading maps
[ Info: 2020-06-17 22:12:18 : Reading maps
[ Info: 2020-06-17 22:12:18 : Reading maps
ERROR: On worker 2:
Base.InvalidCharError{Char}('\xe6')
invalid_char at ./char.jl:85
UInt32 at ./char.jl:130
convert at ./char.jl:180 [inlined]
cconvert at ./essentials.jl:390 [inlined]
lowercase at ./strings/unicode.jl:245
map at ./strings/basic.jl:574
lowercase at ./strings/unicode.jl:531 [inlined]
_guess_file_type at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:153
_ascii_grid_read_header at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:113
_ascii_grid_reader at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:90
read_cellmap at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:68
load_raster_data at /opt/julia/packages/Circuitscape/3Rn8u/src/io.jl:410
raster_pairwise at /opt/julia/packages/Circuitscape/3Rn8u/src/raster/pairwise.jl:18
_compute at /opt/julia/packages/Circuitscape/3Rn8u/src/run.jl:42
macro expansion at ./util.jl:234 [inlined]
compute at /opt/julia/packages/Circuitscape/3Rn8u/src/run.jl:73
#104 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:294
run_work_thunk at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:79
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/process_messages.jl:294 [inlined]
#103 at ./task.jl:358
Stacktrace:
 [1] (::Base.var"#726#728")(::Task) at ./asyncmap.jl:178
 [2] foreach(::Base.var"#726#728", ::Array{Any,1}) at ./abstractarray.jl:1919
 [3] maptwice(::Function, ::Channel{Any}, ::Array{Any,1}, ::Array{Dict{String,String},1}) at ./asyncmap.jl:178
 [4] wrap_n_exec_twice(::Channel{Any}, ::Array{Any,1}, ::Distributed.var"#204#207"{Distributed.WorkerPool}, ::Function, ::Array{Dict{String,String},1}) at ./asyncmap.jl:154
 [5] async_usemap(::Distributed.var"#188#190"{Distributed.var"#188#189#191"{Distributed.WorkerPool,typeof(compute)}}, ::Array{Dict{String,String},1}; ntasks::Function, batch_size::Nothing) at ./asyncmap.jl:103
 [6] #asyncmap#710 at ./asyncmap.jl:81 [inlined]
 [7] pmap(::Function, ::Distributed.WorkerPool, ::Array{Dict{String,String},1}; distributed::Bool, batch_size::Int64, on_error::Nothing, retry_delays::Array{Any,1}, retry_check::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:126
 [8] pmap(::Function, ::Distributed.WorkerPool, ::Array{Dict{String,String},1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:101
 [9] pmap(::Function, ::Array{Dict{String,String},1}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:156
 [10] pmap at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.4/Distributed/src/pmap.jl:156 [inlined]
 [11] compute_metaparallel(::Array{String,1}, ::Int64) at /opt/julia/packages/Circuitscape/z0iGj/src/utils.jl:120
 [12] top-level scope at REPL[12]:1

[vlandau]

could julia recognize that the input is an array when calling compute

I had to create a customized function that accepts arrays for Omniscape. See Omniscape.jl/src/functions.jl#L306. Note that is specifically for advanced mode. You may have to create a similar function for your use case.