Discover this year's submissions to the Dataiku Frontrunner Awards and give kudos to your favorite use cases and success stories!READ MORE

Best way to build partitioned datasets in scenario

Solved!
NN
Neuron
Neuron
Best way to build partitioned datasets in scenario

Hi All,

Suppose i have a flow like image1 below which has 3 partitioned datasets

and i wish to run the flow on a single partition for all 3 datasets using scenario (DSS 8.0.1)

do i have to create a single step for each dataset like shown in image2 below(which means 3 steps for 3 datasets) ?

 
 

 

 

 

0 Kudos
1 Solution
Mattsco
Dataiker
Dataiker

Hi, 

No, you can specify only the last partitioned dataset (weather_conditions_summerized) with the CURRENT_DAY variable. 
But you need to modify the build mode to allow recursivity: "build required" dataset or "force rebuild".
"Build required" will only update the partition if we detect new files in the current day sub path folder of the 1st filesystem dataset. 

Mattsco

View solution in original post

0 Kudos
2 Replies
Mattsco
Dataiker
Dataiker

Hi, 

No, you can specify only the last partitioned dataset (weather_conditions_summerized) with the CURRENT_DAY variable. 
But you need to modify the build mode to allow recursivity: "build required" dataset or "force rebuild".
"Build required" will only update the partition if we detect new files in the current day sub path folder of the 1st filesystem dataset. 

Mattsco
0 Kudos
NN
Neuron
Neuron
Author

Thank you @Mattsco .. That Helps

0 Kudos