Partitions with pyspark and SPARK SQL

dhyadav79
Level 2
Partitions with pyspark and SPARK SQL

Hi,

I have created Dataset and Dataframe in PySpark  and want to create partition using one column. could you please help how can I achieve this.

 

Similarly by using SPARKSQL need to create partition on non date column and that is part of schema, could you please help how can I create partition on it.

0 Kudos
1 Reply
AlexT
Dataiker

Hi,

You may find this doc useful: https://doc.dataiku.com/dss/latest/spark/datasets.html#other

Thanks,

0 Kudos