Ready for Dataiku 10? Try out the Crash Course on new features!GET STARTED

Python recipe with Partitioned input and output

Sajid_Khan
Level 2
Level 2
Python recipe with Partitioned input and output

Hi

Greetings...

I am trying to connect a Python recipe with multiple inputs among which couple of them has to be partitioned so that I can do some transformation on a part of data and then write it to the output dataset.

Things I need to do:

- Partition input dataset on a dimension dynamically. (The list of partition values has to be passed dynamically)

- Read partitioned data into python recipe and do transformations which will be applied on each partitions

- Post transformation, write it to the output dataset.

 

I came through a solution on this community of using scenario to do it. I am not sure how to do that.

Please let me know if any more detail needs to be provided for clarity.

Kindly Help,

Thanks,

Sajid Khan


Operating system used: Windows

0 Kudos
3 Replies
Jurre
Level 4

Hi @Sajid_Khan , 

which community-solution are you referring to ?   The Dataiku Academy has some great resources on partitioning by the way, including a course.  -Jurre

 

0 Kudos
Sajid_Khan
Level 2
Level 2
Author

Hi @Jurre ,

I referred to the Dataiku community discussion. I have gone through the course link which you provided, didn't find anything concrete there. I want to pass the partition lists dynamically.

Thanks for the help

- Sajid

0 Kudos
Jurre
Level 4

Hey @Sajid_Khan ,

It really helps to be specific: what have you tried yourself ? What community solution are you referring to, how have you tried to implement that solution ? 

There are a lot of resources here to help you, for example :

I hope these resources will hand you some clues on where to start and how to achieve your objective. When posting back please be as specific as possible and include what you have tried yourself. This article is very informative on that matter. Happy coding!

Edit : added link to community resource

0 Kudos

Labels

?
Labels (4)

Setup info

?
A banner prompting to get Dataiku DSS