Incompatibilities b/w pandas and pySpark

[thepushkarp]

pandas and pySpark DataFrames do not have the same functions to perform the operations in their dataframes, because they both read and process data differently. We should document these incompatibilities to know where we might have to change prior (and future) implementations.

[thepushkarp]

Most differences arise between the two because of the way both dataframes are made. Here are some of the ways they both differ:

• pandas dataframes are mutable and work on a single machine, while pySpark dataframes are immutable and support parallel distributed workloads
• pandas dataframes operate natively with python while pySpark dataframes operate on JVM
• pandas dataframes execute non-lazily, while pySpark dataframes execute lazily

Essentially, they both are very different datatypes despite having similar names.

There is also a native API provided by pySpark to offer some compatibility with the pandas syntax: Pandas API on Spark