README.rst

Spark Partitionr

This project aims at making it easy to load a dataset supported by Spark and create a Hive table partitioned by a specific column. The output is written using one of the output format supported by Spark.

Usage

To use it

>>> import pyspark.sql.functions as sf
>>> column = 'a_column_with_unixtime'
>>> partition_function = lambda column: sf.from_unixtime(sf.col(column), fmt='yyyy-MM-dd')

>>> from spark_partitionr import main
>>> main('hdfs:///data/some_data', format_output='parquet',
         database='my_db', table='my_tbl', mode_output='overwrite',
         partition_col='dt', partition_with=partition_function('a_col'),
         master='yarn', format='com.databricks.spark.csv',
         header=True)

There are a number of additional arguments, among which spark=spark if you want to pass your own SparkSession (very helpful when you need to process lots of datasets this way)