append table

Options
Oh_Lily
Oh_Lily Dataiku DSS Core Designer, Registered Posts: 12

Hello,
I have an output table in my dataiky flow where results are appended after each build.
We had a data problem and we would like to relaunch the last run while removing the results of the previous one from the table and replacing it by the new one?
How can we do this without revert to a previous version because we have other changes we would like to keep.

Thank you in advance for your help


Operating system used: windows

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,740 Neuron
    Options

    There is no built-in functionality for what you want to do. Furthermore I would warn you that append recipes are risky thing to use in Dataiku if you depend on your historical data. The default functionality in Dataiku is that when there are schema changes and changes are propagated is to drop and recreate the table, irrespective of you having set the "Append instead of override" check box in any recipe.

  • Oh_Lily
    Oh_Lily Dataiku DSS Core Designer, Registered Posts: 12
    Options

    Thank you for your response !

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,740 Neuron
    Options

    A better way of doing this will be to take the output of the Dataiku flow and do deltas outside of Dataiku so that you can manage your historical table outside of Dataiku. If you need to read your historical table in Dataiku you can add it to the flow as "unmanaged/external" dataset which means Dataiku will not be responsible for managing the schema of that table and you will need to manage it yourself (ie adding/deleting columns, etc).

Setup Info
    Tags
      Help me…