Standardizing Weld's IR code

[espetro]

Hi there,

I have a growing interest in this project and, as it's asked in other issues, I don't know if this project is still active or not. That being said, I've seen that the current implementation seizes the power of Rust, but what about using IR frameworks such as MLIR ?

[sppalkia]

Good question, MLIR is indeed interesting for Weld's use case. We haven't tried it before (and we're currently not working on Weld actively), but if someone were to take a stab at using it as Weld's backend, I'd be happy to provide reviews/suggestions.

The Weld frontend currently doesn't have too many optimizations around n-ary arrays (and in fact, does not yet support a tensor type since the primary use case for now has been more 'slice and dice' ETL-like workloads in libraries like Pandas), so that may be a good starting point: if the frontend is capable of applying some interesting optimizations over tensors, the availability of these types in the backend IR will become much more compelling.