Announcing the winners & finalists of the Dataiku Frontrunner Awards 2021! Read their inspiring stories

Unable to write spark df to csv with column headers and multiple partitions?

parul
Level 1
Unable to write spark df to csv with column headers and multiple partitions?
Writing spark df into csv along with headers repartitions the df into 1 by default. So it takes a lot of time while writing considering the dataset is large because only 1 partition is active. How do I write spark dataframe to csv in hdfs with column headers and multiple partitions, so that it runs faster?
0 Kudos
0 Replies
Labels (3)
A banner prompting to get Dataiku DSS