Is there a simple way to reapply flow/recipes to a second dataset?

UserBird
Dataiker
Is there a simple way to reapply flow/recipes to a second dataset?
Hi

I created a flow sequencing recipes for a first dataset. The goal is to create a prediction model at the end of the flow.

Next I need to apply my model to an input dataset that has the same schema as my first dataset.

I could not figure out how to apply the whole flow to the 2nd dataset and I had to copy the flow steps (analysis, joins, python) and apply them to the 2nd dataset.

Is there a more straight forward way? The issue is that if I add a new step or modify a recipe to enrich the dataset, it must be done twice.

Best regards

Geoff
7 Replies
jereze
Community Manager
Community Manager
Hi,

This is a good question. There is no out-of-the-box feature to do that today. But we are thinking about it.

Actually, it works in a simple case since v2.0.0: when you have your data preparation in a script of an Analysis (called "Analyse"), deploying the model will also reproduce the script (ie. all the processors).

If you have more a complex flow (with Python recipes, etc.), there is a solution: you can stack your two sources in a single dataset, then apply your flow of transformation, then split in two before modelling.

Jeremy
Jeremy, Product Manager at Dataiku
okiriza
Level 2
Hi, I am also interested in this feature. Is there any support for it in Dataiku v2.3?

Thanks.
0 Kudos
bored_panda
Level 2
Isn't it possible to export your project (project home page > actions > export this project), reimport it (DSS home page > mouse on the left > import) with a different name, and then change the input dataset ?
0 Kudos
tjh
Level 3
This leads however to very strange behaviour in the new project
0 Kudos
biodan25
Level 1
Is there any update ? I've downloaded DSS 4.3.1 to evaluate whether it will support a project that will have serial data tables that will be inserted as records in a database over time. I'd like to run a code recipe on each new 'record'. Note that a database record will reflect a file/table containing multiple columns and rows. Is this possible currently?
0 Kudos
asdfasdf
Level 1

Iโ€™m doing timeseries demand forecasting on ~10K products, with distinct behaviors. So one model per product. 

In Spark Iโ€™d just do a groupby(productID).apply(modelCode).

 

Whatโ€™s an efficient to code, efficient to run way to do this in pandas in Dataiku? 

Best to do partitioned model? The data is sitting on Snowflake, so for partitioning, do I cluster on productID? (Actually products are identified over 5 features). 

0 Kudos
Turribeach

Please post a new thread. This thread is 8 years old.

0 Kudos