Discover this year's submissions to the Dataiku Frontrunner Awards and give kudos to your favorite use cases and success stories!READ MORE

Unable to write spark df to csv with column headers and multiple partitions?

parul
Level 1
Unable to write spark df to csv with column headers and multiple partitions?
Writing spark df into csv along with headers repartitions the df into 1 by default. So it takes a lot of time while writing considering the dataset is large because only 1 partition is active. How do I write spark dataframe to csv in hdfs with column headers and multiple partitions, so that it runs faster?
0 Kudos
0 Replies

Labels

?
Labels (3)
A banner prompting to get Dataiku