Python recipe with Partitioned input and output

Options
Sajid_Khan
Sajid_Khan Partner, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 12 Partner

Hi

Greetings...

I am trying to connect a Python recipe with multiple inputs among which couple of them has to be partitioned so that I can do some transformation on a part of data and then write it to the output dataset.

Things I need to do:

- Partition input dataset on a dimension dynamically. (The list of partition values has to be passed dynamically)

- Read partitioned data into python recipe and do transformations which will be applied on each partitions

- Post transformation, write it to the output dataset.

I came through a solution on this community of using scenario to do it. I am not sure how to do that.

Please let me know if any more detail needs to be provided for clarity.

Kindly Help,

Thanks,

Sajid Khan


Operating system used: Windows

Answers

  • Jurre
    Jurre Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Registered, Dataiku DSS Developer, Neuron 2022 Posts: 114 ✭✭✭✭✭✭✭
    Options

    Hi @Sajid_Khan
    ,

    which community-solution are you referring to ? The Dataiku Academy has some great resources on partitioning by the way, including a course. -Jurre

  • Sajid_Khan
    Sajid_Khan Partner, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 12 Partner
    Options

    Hi @Jurre
    ,

    I referred to the Dataiku community discussion. I have gone through the course link which you provided, didn't find anything concrete there. I want to pass the partition lists dynamically.

    Thanks for the help

    - Sajid

  • Jurre
    Jurre Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Registered, Dataiku DSS Developer, Neuron 2022 Posts: 114 ✭✭✭✭✭✭✭
    Options

    Hey @Sajid_Khan
    ,

    It really helps to be specific: what have you tried yourself ? What community solution are you referring to, how have you tried to implement that solution ?

    There are a lot of resources here to help you, for example :

    I hope these resources will hand you some clues on where to start and how to achieve your objective. When posting back please be as specific as possible and include what you have tried yourself. This article is very informative on that matter. Happy coding!

    Edit : added link to community resource

Setup Info
    Tags
      Help me…