Asynchronous web tasks

[daquinteroflex]

When doing parameter scans, especially with large simulation tasks like those that use CustomMediums, it takes a pretty long time to create/upload the task, and then sequentially download it. Could we paralellize this process?

Note related to #1242 but specifically on asynchronous tasks.

This is however a bit broader:

For example, in https://docs.flexcompute.com/projects/tidy3d/en/latest/notebooks/MetalHeaterPhaseShifter.html

for _, hs_data in batch_data.items():
    temp_interpolated = hs_data["temperature"].temperature.interp(x=target_grid.x, y=0, z=target_grid.z, fill_value=300)
    psim = optic_sim.perturbed_mediums_copy(temperature=temp_interpolated)
    perturb_sims.append(psim)

It'd be nice if hs_data could be just a collection of task result references, that can be downloaded, and not have to be uploaded to the cloud again.

Useful reading:

• https://realpython.com/async-io-python/
• https://realpython.com/python-concurrency/
• https://blog.jupyter.org/ipython-7-0-async-repl-a35ce050f7f7
• Official docs https://docs.python.org/3/library/asyncio.html
• Handy guide https://superfastpython.com/asyncio-for-loop/
• https://stackoverflow.com/questions/28492103/how-to-combine-python-asyncio-with-threads

[daquinteroflex]

Theory behind the implementation options

We want users to access asynchronous commands without much complexity. In my opinion, we want:

• To support both concurrent and parallel tasks framework
• Make it easy for the user to interact with these aynchronous commands without necessarily having to understand the asynchronous packages underneath.

From theiron.io blog:

Candidate implementation packages suggested:

• https://docs.python.org/3/library/asyncio.html
• https://docs.python.org/3/library/multiprocessing.html
• https://docs.python.org/3/library/threading.html#module-threading
• https://docs.python.org/3/library/threading.html
• https://docs.python.org/3/library/functools.html

Important concepts to understand:

• IO-bound process https://en.wikipedia.org/wiki/I/O_bound
• CPU-bound process https://en.wikipedia.org/wiki/CPU-bound
• https://en.wikipedia.org/wiki/Proxy_server
• https://stackoverflow.com/questions/2846653/how-do-i-use-threading-in-python

CPU-bound jobs will spend most of their execution time on actual computation ("number crunching"[1]) as opposed to e.g. communicating with and waiting for peripherals such as network or storage devices (which would make them I/O bound instead).

In our case, our blocking functions are pretty clear: the user has to await our server to receive the uploaded simulation, run the pipeline, and download the simulation. In this sense, fundamentally our web api has to wait for such operations to be completed, and hence our operations are mainly IO-limited really.

Now, let's evaluate each package according to this requirement:

"asyncio is often a perfect fit for IO-bound and high-level structured network code"

I have also looked into multiprocess. The documentation and version management is not great in my opinion https://multiprocess.readthedocs.io/en/latest/multiprocess.html

Requirements

One of the main things to define is what we want to parallelise and what we don't.

My personal requirements based on my understanding are for the 3.0 architecture:

• Parallelise uploading and running tasks up as limited by the user bandwidth
  • Be able to manage and keep track of these tasks asynchronously

Implementation caveats

• By default asyncio works based on one thread. Ideally, we want to leverage multiple CPU threads and all our internet bandwidth to parallelize our operations.