Best way to build partitioned datasets in scenario

NN
NN Neuron, Registered, Neuron 2022, Neuron 2023 Posts: 145 Neuron

Hi All,

Suppose i have a flow like image1 below which has 3 partitioned datasets

and i wish to run the flow on a single partition for all 3 datasets using scenario (DSS 8.0.1)

do i have to create a single step for each dataset like shown in image2 below(which means 3 steps for 3 datasets) ?

image

image

Best Answer

  • Mattsco
    Mattsco Dataiker, Registered Posts: 125 Dataiker
    Answer ✓

    Hi,

    No, you can specify only the last partitioned dataset (weather_conditions_summerized) with the CURRENT_DAY variable.
    But you need to modify the build mode to allow recursivity: "build required" dataset or "force rebuild".
    "Build required" will only update the partition if we detect new files in the current day sub path folder of the 1st filesystem dataset.

Answers

Setup Info
    Tags
      Help me…