how can select the append mode in a dataset?

Registered Posts: 10 ✭✭✭✭
I want to append data everyday after to manipulate it.

Welcome!

It looks like you're new here. Sign in or register to get started.

Answers

  • Registered, Neuron 2022 Posts: 121 ✭✭✭✭✭✭

    Some datasets supports append, some not. Note that this mode is only available on recipe with output dataset using an infrastructure allowing append (e.g. it is not possible with HDFS).



    Or you can use partitioning and put new data every day into a new partition.

    First you need to activate partitioning on the dataset. You need to choose between time based or discrete partitioning. If you choose time base partitioning, then you can partition by year, year+month, or year+month+day.

    Or you can partition using any discrete value, for example you choose partitioning by City, or by Country.

    Then when you have activated the partitioning on the dataset, the icon should change.

    After that you need to enter into the parent recipe and define the partition range what you want to calculate:

  • Registered Posts: 10 ✭✭✭✭
    What are the datasets that support append?
    How can put new data into a new partition?
    Sorry, I'm noob in Dataiku
  • Registered, Neuron 2022 Posts: 121 ✭✭✭✭✭✭
    see my updated answer.
  • Registered Posts: 10 ✭✭✭✭
    Hi, and what are the datasets that support append?
    I want to schedule my flow everyday, and the data result of my flow I want to append with the previous data result, because I must to create a report with this data.
    Thanks

Welcome!

It looks like you're new here. Sign in or register to get started.

Welcome!

It looks like you're new here. Sign in or register to get started.