Discover the winners & finalists of the 2022 Dataiku Frontrunner Awards!READ THEIR USE CASES

Best way to build partitioned datasets in scenario

Solved!
NN
Best way to build partitioned datasets in scenario

Hi All,

Suppose i have a flow like image1 below which has 3 partitioned datasets

and i wish to run the flow on a single partition for all 3 datasets using scenario (DSS 8.0.1)

do i have to create a single step for each dataset like shown in image2 below(which means 3 steps for 3 datasets) ?

 
 

 

 

 

0 Kudos
1 Solution
Mattsco
Dataiker

Hi, 

No, you can specify only the last partitioned dataset (weather_conditions_summerized) with the CURRENT_DAY variable. 
But you need to modify the build mode to allow recursivity: "build required" dataset or "force rebuild".
"Build required" will only update the partition if we detect new files in the current day sub path folder of the 1st filesystem dataset. 

Mattsco

View solution in original post

0 Kudos
2 Replies
Mattsco
Dataiker

Hi, 

No, you can specify only the last partitioned dataset (weather_conditions_summerized) with the CURRENT_DAY variable. 
But you need to modify the build mode to allow recursivity: "build required" dataset or "force rebuild".
"Build required" will only update the partition if we detect new files in the current day sub path folder of the 1st filesystem dataset. 

Mattsco
0 Kudos
NN
Author

Thank you @Mattsco .. That Helps

0 Kudos