The Dataiku Frontrunner Awards have just launched to recognize your achievements! Submit Your Entry

Best way to build partitioned datasets in scenario

Solved!
NN
Level 3
Best way to build partitioned datasets in scenario

Hi All,

Suppose i have a flow like image1 below which has 3 partitioned datasets

and i wish to run the flow on a single partition for all 3 datasets using scenario (DSS 8.0.1)

do i have to create a single step for each dataset like shown in image2 below(which means 3 steps for 3 datasets) ?

 
 

 

 

 

0 Kudos
1 Solution
Mattsco
Dataiker
Dataiker

Hi, 

No, you can specify only the last partitioned dataset (weather_conditions_summerized) with the CURRENT_DAY variable. 
But you need to modify the build mode to allow recursivity: "build required" dataset or "force rebuild".
"Build required" will only update the partition if we detect new files in the current day sub path folder of the 1st filesystem dataset. 

Mattsco

View solution in original post

0 Kudos
2 Replies
Mattsco
Dataiker
Dataiker

Hi, 

No, you can specify only the last partitioned dataset (weather_conditions_summerized) with the CURRENT_DAY variable. 
But you need to modify the build mode to allow recursivity: "build required" dataset or "force rebuild".
"Build required" will only update the partition if we detect new files in the current day sub path folder of the 1st filesystem dataset. 

Mattsco

View solution in original post

0 Kudos
NN
Level 3
Author

Thank you @Mattsco .. That Helps

0 Kudos
A banner prompting to get Dataiku DSS