Allow custom copy to output in application-as-recipe

Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 317 Neuron

Currently an application-as-recipe copies the result to the output table in what is essentially a sync step as noted in this post. There is a good rationale for this approach as it accommodates the full range of data connection types. However, this approach prevents an application-as-recipe from running entirely in database and thus precludes use cases in which only in database execution is practical.

The idea is to provide a setting for an application-as-recipe that leaves the copy step up to the application developer. The application developer would then specify a final Execute Python step in the application scenario that handles the copy to the output dataset. Being able to specify this final Python step would enable the application developer to make the copy to output run in database. This approach extends the one described in this post.

The application developer would be responsible for either assuring that the application-as-recipe is used in expected ways (e.g., all steps run on the same database platform) or otherwise handling the variations that might come up.

cc @pvannies
, @akshaykatre
, @fchataigner2

2 votes

New · Last Updated


  • pvannies
    pvannies Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Frontrunner 2022 Participant, Neuron 2023 Posts: 16 Neuron

    Fully support this idea. We have discussed with dataiku that the remapping of the connections could happen automatically as well (based on the input dataset). But having the final step up to the application developer indeed solves the problem for the last copy/sync step of the output dataset.

Setup Info
      Help me…