Application-as-recipe

subhash

I'm new to dataiku & looking for some help in building a feature that can be reused. I'm looking ways to build a component that I can use for capturing some metrics across multiple flows.

While using application-as-recipe, i followed the tutorials mentioned below, but the project i created as application is not visible in the recipes to add. Any permissions that i'm missing? I have admin privileges to dataiku.

https://knowledge.dataiku.com/latest/courses/o16n/dataiku-applications/create-app-as-recipe.html

Also, is there any other documentation or links that can help me understand how we can have dynamic datasets as input and single application to basically use the dataset and capture some metrics into s3 or some database?

KimmyC

Hi Subhash,

Did you give the app-as-recipe a category? If so, could you please try restarting DSS and see if it's now available under +Recipes? If you did not give it a category, then it should be available under +Recipes > Applications section.

As for your second question, do you mind elaborating further on what you mean by dynamic datasets? Technically you can configure it to use any input dataset within your project, assuming that it shares the same schema as the input recipe for your application-as-recipe. We do not have specific documentation on this, however you could consider looking into our tutorial on this particular feature. Additionally, I've found another community thread that might be useful for this topic.

Hope this helps,

Kim

subhash

Thanks @KimmyC
I'll go over the documentation. Here's the gist of what I'm trying to achieve

We have multiple projects with different flows doing ETL jobs. Requirement is to capture some stats at each stage in the project and log/store in s3. I want to have common application that can be used across projects.

Since you mention the schema of input dataset has to be same, I doubt if my solution can have same schema for all the datasets across multiple projects. Instead, can I take advantage of dataiku properties/variables as input or use json as input for the application-as-recipe, do some transformation and write to dataset in S3? Any thoughts on this approach?

KimmyC

Hi Subhash,

Thank you for clarifying a bit more. At this point, it'll be more effective if we can understand your use case further with a technical resource from Dataiku. I've reached out separately to coordinate the next steps.

Thanks,

Kim