Partitions with pyspark and SPARK SQL
Dheerendra
Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered Posts: 21 ✭✭✭✭
Hi,
I have created Dataset and Dataframe in PySpark and want to create partition using one column. could you please help how can I achieve this.
Similarly by using SPARKSQL need to create partition on non date column and that is part of schema, could you please help how can I create partition on it.
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,225 DataikerHi,
You may find this doc useful: https://doc.dataiku.com/dss/latest/spark/datasets.html#other
Thanks,