Does Ballista have any form of caching for queries on S3 files?

[collimarco]

I wonder if Ballista can cache (in memory or on disk) the files (or parts of the files) that are downloaded from S3.

For example, if I run the same query or a similar query on the same dataset of Parquet files stored on S3, does Ballista downloads the files every time or it has some form of caching (so that the next queries are faster and less downloads are required)?

[mingmwang]

@collimarco
Currently in the main branch there is no.
@yahoNanJing had implemented an adaptive data cache in his own branch and I think he will contribute that to the main branch soon.

[yahoNanJing]

Hi @collimarco, we have implemented the data cache for the Ballista on my personal branch. It depends on another PR heavily for the cache aware task scheduling #823. After #823, I will raise another PR to contribute the data cache to the main branch.

If you are interested on this feature, you can follow #645