Can application-as-recipe write "in database" to SQL output dataset?

Marlan
Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 319 Neuron

Hello,

I have a number of use cases for the application-as-recipe feature but after some initial testing, it appeared that this feature didn't work with SQL Script recipes and SQL Datasets. However it looks like an application-as-recipe can - with some Python supplementation - work with SQL Script recipes as described on the Community in this post and in this post. Thank you @akshaykatre
, @pvannies
, and @fchataigner2
for your questions and solutions on this.

It's still unclear to me though whether the final step of populating the output dataset (in the calling project) will or can run in database? My impression is that this final step is essentially a synch recipe. That is, DSS does the copy from the original output table to the calling project output table. If so, this would limit our use cases as what would normally take a few seconds in SQL would take quite a bit longer to copy through DSS.

Can anyone provide more specifics on how the "copy" to the output table is done? And specifically, is there a way to have this run in database?

Thanks,

Marlan

Best Answer

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker
    Answer ✓

    Hi,

    you're correct, the final "copying back of the application outputs to the recipe original outputs" is essentially a Sync recipe, and it will stream the data via the backend, not run in-database. This was decided to keep it simple in face of the many possible combination of dataset to copy from -> dataset to copy to.

Answers

  • Marlan
    Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 319 Neuron

    OK thanks @fchataigner2
    . Not what I was hoping to hear but certainly understandable as to why it works that way.

    Marlan

Setup Info
    Tags
      Help me…