Memory optimization across multiple devices

[zealbell]

Today, I stumbled upon withBatch() from the repo. From the docs it says

• withBatch()-> Enable batch processing. TornadoVM will… This is used mainly when users want to execute big data applications that do not fit on the device’s global memory

I get the explanation. I just want to know more about how execution works across multiple devices especially when there are memory constraints. Can a single execution plan (TornadoExecutionPlan) employ multiple devices during execution? Or do developers have to implement multiple graphs/plans asper the number of devices they require?

Hypothetically speaking; here’s what I mean, say I am trying to accelerate a method whose input array is worth 100GB on a fleet 10 RTX 3080 cards. Since memory on each card is about 11GB+, what would Tornado do if I created a single execution plan withBatch(“10240MB”). Would execution run only on one card with data batched at 10GB or would it share batched data across all my cards simultaneously

[jjfumero]

Hi @54LiNKeR ,
this is an interesting question, and I think (from the TornadoVM side) we need to improve this documentation. My answers below:

I get the explanation. I just want to know more about how execution works across multiple devices especially when there are memory constraints. Can a single execution plan (TornadoExecutionPlan) employ multiple devices during execution? Or do developers have to implement multiple graphs/plans asper the number of devices they require?

As we have it in v0.15, TornadoVM execution plans run on a single device. There is no automatically multi-device support. Thus, the batch size affects to the buffer size assigned to the execution plan for the target accelerator (target GPU/FPGA/CPU).

Hypothetically speaking; here’s what I mean, say I am trying to accelerate a method whose input array is worth 100GB on a fleet 10 RTX 3080 cards. Since memory on each card is about 11GB+, what would Tornado do if I created a single execution plan withBatch(“10240MB”). Would execution run only on one card with data batched at 10GB or would it share batched data across all my cards simultaneously

This case will be very interesting. So, as I mention, actions set to execution plans affect a single device. This could be a nice extension for the future of TornadoVM.

[zealbell]

Awesome! I look forward to seeing this feature implemented in the future