Added Modin for return of a block

[CLCG-Data-Engineering]

Description

Added Modin pandas DataFrame as type for accepting in return of a block. This way a developer doesn't need to manually move it to a pandas DF and also allows them to load/write data in a distributed way.

Able to read the output from the parent block and keep the data type as Modin.

How Has This Been Tested?

• Created unit tests in the test_variable.py that are passing.
• Tested through the UI

Checklist

• The PR is tagged with proper labels (bug, enhancement, feature, documentation)
• I have performed a self-review of my own code
• I have added unit tests that prove my fix is effective or that my feature works
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• If new documentation has been added, relative paths have been added to the appropriate section of docs/mint.json

cc:

[CLCG-Data-Engineering]

Some questions, since not everything is working as intended yet.

I tried to follow all the steps that are being done with Polars & Pandas, for writing, reading, and deleting files.

1. Output files are somehow not being deleted, and I am unsure where I need to make a change/modify my changes (Running the pipeline twice results in: Is a directory: '/root/.mage_data/default_repo/pipelines/effortless_feather/.variables/load_titanic/output_0/data.parquet'
2. Should I try to downgrade the requirement, to retry the test_backend (3.8) even if it could be at the cost of the level of Pandas API coverage on Modin?

[wangxiaoyou1993]

Hi @CLCG-Data-Engineering , do you still have the issues or questions on this PR?

[CLCG-Data-Engineering]

Yes, Modin is creating a folder with parquet files, and outside that I should test if the behavior to get to a single parquet file, instead of a folder, is possible in the same way, with both Ray and Dask.

But so far I didn't have time to look at it again.

It's definitely still something I would love to have integrated inside of Mageai :)