Dynamically generate output datasets from a Python recipe

kirilov
kirilov Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 4

Hi everyone,

I am trying to implement a Python recipe which should output multiple datasets, based on the logic inside of the recipe. I couldn't find anything like this in the docs, so I was wondering if you have some ideas on how to achieve this behaviour? The idea is that the code would produce a large number of datasets from different dataframes, so doing it manually is not an option.

Thanks in advance!

Tagged:

Answers

  • HarizoR
    HarizoR Dataiker, Alpha Tester, Registered Posts: 138 Dataiker

    Hi,

    It is generally not a good practice to have a "dynamic" number of output Datasets. Do you have simple examples of how the data would be split and the underlying logic? Depending on the use-case, maybe partitioning can help.

    Best,

    Harizo

  • kirilov
    kirilov Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 4

    Hi Harizo,

    Thanks for the insight. I understand that it's probably not a good practice, but I've intentionally avoided the partitioning approach due to other complexities within our project and data situation. That's why I wanted to check if you have any other tips or ideas.

    Kind regards,
    Martin

Setup Info
    Tags
      Help me…