Automation of the cross validation with several recipes to be executed

MaxenceQueyrel Registered Posts: 3 ✭✭✭


I run a flow that takes as input a dataset that is split by a python recipe into training and validation sets.
Then the training and validation sets go through several recipes (custom or not) until the results are saved in a folder.
My question is, how can I automate a cross validation with my split recipe? For example, I would like to use different predefined seeds or splits and run the flow multiple times until all tests are completed. I am attaching a screenshot of my flow, with the split recipe on the left, the calculation recipes on top or not shown (in another views) and the last recipe to save the result in my folder.

Thank you for your help,



Best Answer

  • StanG
    StanG Dataiker, Registered Posts: 52 Dataiker
    edited July 17 Answer ✓

    Hi Maxence,
    You can define project variables (for your seed and number of splits in your case) and retrieve these variables in the python recipe using:


    Then you can create a python script (in a scenario for example) that will build the flow multiple times with different values for the variables with something like:

    project = dataiku.api_client().get_project(dataiku.default_project_key())
    dataset_to_build = project.get_dataset(YOUR_DATASET)
    for seed in seeds:
        project_variables = project.get_variables()
        project_variables['standard']['seed'] = seed


Setup Info
      Help me…