Dynamically generate output datasets from a Python recipe

kirilov

Hi everyone,

I am trying to implement a Python recipe which should output multiple datasets, based on the logic inside of the recipe. I couldn't find anything like this in the docs, so I was wondering if you have some ideas on how to achieve this behaviour? The idea is that the code would produce a large number of datasets from different dataframes, so doing it manually is not an option.

Thanks in advance!

HarizoR

Hi,

It is generally not a good practice to have a "dynamic" number of output Datasets. Do you have simple examples of how the data would be split and the underlying logic? Depending on the use-case, maybe partitioning can help.

Best,

Harizo

kirilov

Hi Harizo,

Thanks for the insight. I understand that it's probably not a good practice, but I've intentionally avoided the partitioning approach due to other complexities within our project and data situation. That's why I wanted to check if you have any other tips or ideas.

Kind regards,
Martin