[Question] ArrayRDD to Pyspark Dataframe?

[osimpson]

Hi - thanks so much for this package!

I came to this repo because I need to run a scikit-learn predictive model on Spark. It is easy to map the model with ArrayRDDs. However, my postprocessing assumes a PySpark DataFrame. Is there a way to convert an ArrayRDD to a DataFrame?

I appreciate any help, thanks!

[sajjadGG]

if its size isn't big I think you can collect it and convert it to a list and then create your data frame. (you can call collect or tolist on ArrayRDD) however I believe there are more efficient options