add parallelization

[bnord]

No description provided.

[bnord]

Notes and Ideas on how to implement parallelization

Likely key components

1. map() function
2. import multiprocessing as mppython

workers = mp.cpu_count() - 1
with mp.Pool(workers) as p:

4. example:

import multiprocessing as mp
import random
import time

def some_calculations(x: float) -> float:
    time.sleep(.5)
    # some calculations are done
    return x**2

if __name__ == "__main__":
    x_list = [random.random() for _ in range(20)]
    with mp.Pool(4) as p:
        x_recalculated = p.map(some_calculations, x_list)

5. joblib
6. asyncio
7. example:

import multiprocessing
from joblib import Parallel, delayed
from tqdm import tqdm

num_cores = multiprocessing.cpu_count()
inputs = tqdm(myList)

if __name__ == "__main__":
    processed_list = Parallel(n_jobs=num_cores)(delayed(myfunction)(i,parameters) for i in inputs)

Sources

1. https://towardsdatascience.com/parallelization-in-python-the-easy-way-aa03ed04c209
2. https://stackoverflow.com/questions/9786102/how-do-i-parallelize-a-simple-python-loop
3. https://medium.com/@mjschillawski/quick-and-easy-parallelization-in-python-32cb9027e490