multiprocessing consistency across cooltools

[gfudenberg]

many user-facing functions take nproc as an argument.
e.g.

cooltools/cooltools/api/snipping.py

Line 824 in c85d647

def pileup(

cooltools.sample() does not

cooltools/cooltools/api/sample.py

Line 54 in c85d647

map_func=map,

(however, multiprocessing is offered as an nproc argument in=cooltools.cli.random-sample does)

Any suggestion on preferred behavior?

[Phlya]

Can we make this into a decorator to ensure the argument is consistent across all tools and stays like that in the future?

[gfudenberg]

note 10/23/2023: some of the cootlools functions using pool.map should be changed to pool.imap!

see @nvictus comment #464 (comment)

[gfudenberg]

in the process of addressing this issue, @Yaoyx ran into a design choice:
(a) keep multiprocessing & factor out internal lambda functions (which pickle fails on)
(b) move to multiprocess (which uses dill instead of pickle) and otherwise keep code the same

any recommendations @golobor @nvictus @agalitsyna @sergpolly ?
an example of an internal lambda function that would need to be moved is here, inside of dots scoring_and_histogramming():

job = lambda tile: to_hist(to_score(tile))

cooltools/cooltools/api/dotfinder.py

Line 1307 in 81b9045

job = lambda tile: to_hist(to_score(tile))

[sergpolly]

imho multiprocess/dill is much more preferable, because multiprocessing/pickle prevents one more thing: open2c/open2c_examples#29

also I've been trying to hint at that here #477

[nvictus]

Multiprocessing will eventually let you register custom serializers, but until that happens I would stick with multiprocess+dill or another 3rd party pool manager like loky or mpire.

Also relevant: python/cpython#89467

[nvictus]

We might also want to explore changing the default context (which is "fork" on linux) to:

from multiprocess import get_context

with get_context("spawn").Pool() as pool:
    ....

See this for details.

[gfudenberg]

closed in #489