SQL on partitioned table - how to launch computation on ALL partitions

@tanguy
,

There is no option to build a dataset or run a recipe on all partitions. We aware of this and we have captured this in our backlog. For the time being what you can do to build all partitions:

1) With time range dimension you can use the date range directly in the recipe or scenario, in an example where you have a monthly partition, you can use: e.g 2021-01 / 2021-12. If you want to know when your partitions start/end you can explore the Dataset and under Status, you will be able to sort asc/desc to determine this.

2) For discreet dimensions you can use Python steps in a scenario with an execute python code step:

from dataiku.scenario import Scenario
import dataiku

#replace dataset name here

dataset_name = "bulldozer_small_extract_copy"

scenario = Scenario()
dataset = dataiku.Dataset(dataset_name)
partitions = dataset.list_partitions() # get all partitions from input dataset

# for all available partitions in all dimensions 
partitions_str = ','.join(partitions) # concatenate 

print("all partitions are:", partitions_str)
#when some dimensions are defined but another dimensions requires ALL include in your example partitions you want will start with '2020Q4|Pricing|L4L_Monthly'
#partitions_str = ','.join([item for item in partitions if item.startswith('2020Q4|Pricing|L4L_Monthly')])

scenario.build_dataset(dataset_name, partitions=partitions_str)

Let me know if this works for you.