Announcing the winners & finalists of the Dataiku Frontrunner Awards 2021! Read their inspiring stories

Can application-as-recipe write "in database" to SQL output dataset?

Solved!
Marlan
Neuron
Neuron
Can application-as-recipe write "in database" to SQL output dataset?

Hello,

I have a number of use cases for the application-as-recipe feature but after some initial testing, it appeared that this feature didn't work with SQL Script recipes and SQL Datasets. However it looks like an application-as-recipe can - with some Python supplementation - work with SQL Script recipes as described on the Community in this post and in this post. Thank you @akshaykatre@pvannies, and @fchataigner2 for your questions and solutions on this.

It's still unclear to me though whether the final step of populating the output dataset (in the calling project) will or can run in database? My impression is that this final step is essentially a synch recipe. That is, DSS does the copy from the original output table to the calling project output table. If so, this would limit our use cases as what would normally take a few seconds in SQL would take quite a bit longer to copy through DSS.

Can anyone provide more specifics on how the "copy" to the output table is done? And specifically, is there a way to have this run in database?

Thanks,

Marlan

 

0 Kudos
1 Solution
fchataigner2
Dataiker
Dataiker

Hi,

you're correct, the final "copying back of the application outputs to the recipe original outputs" is essentially a Sync recipe, and it will stream the data via the backend, not run in-database. This was decided to keep it simple in face of the many possible combination of dataset to copy from -> dataset to copy to.

View solution in original post

0 Kudos
2 Replies
fchataigner2
Dataiker
Dataiker

Hi,

you're correct, the final "copying back of the application outputs to the recipe original outputs" is essentially a Sync recipe, and it will stream the data via the backend, not run in-database. This was decided to keep it simple in face of the many possible combination of dataset to copy from -> dataset to copy to.

View solution in original post

0 Kudos
Marlan
Neuron
Neuron
Author

OK thanks @fchataigner2. Not what I was hoping to hear but certainly understandable as to why it works that way.

Marlan

0 Kudos
A banner prompting to get Dataiku DSS