Join us on July 16th as we explore real-world Reinforcement Learning Learn more

how can select the append mode in a dataset?

Level 1
how can select the append mode in a dataset?
I want to append data everyday after to manipulate it.
0 Kudos
4 Replies
Level 4

Some datasets supports append, some not.  Note that this mode is only available on recipe with output dataset using an infrastructure allowing append (e.g. it is not possible with HDFS).



Or you can use partitioning and put new data every day into a new partition.



First you need to activate partitioning on the dataset. You need to choose between time based or discrete partitioning. If you choose time base partitioning, then you can partition by year, year+month, or year+month+day.



Or you can partition using any discrete value, for example you choose partitioning by City, or by Country.





Then when you have activated the partitioning on the dataset, the icon should change.



After that you need to enter into the parent recipe and define the partition range what you want to calculate:



 

0 Kudos
Level 1
Author
What are the datasets that support append?
How can put new data into a new partition?
Sorry, I'm noob in Dataiku
0 Kudos
Level 4
see my updated answer.
0 Kudos
Level 1
Author
Hi, and what are the datasets that support append?
I want to schedule my flow everyday, and the data result of my flow I want to append with the previous data result, because I must to create a report with this data.
Thanks
0 Kudos
Labels (2)