Cannot use a managed folder as an input of a python recipe created from python code using builder

AnnaZ
AnnaZ Dataiku DSS Core Designer, Registered Posts: 3

Hi team,

I currently encountered an urgent issue with our project automation pipeline. In a Python code recipe, I used builder to create a new python recipe, and have added a dataset as input, 3 new datasets as output. In order to save some model objects to a managed folder, I need to add an existing folder "Pickle_Files" as input of the newly created python recipe. (I can't add the folder through the UI, because all the steps and files are automatically generated by the previous python script).

I didn't have issues when I used files as inputs. So is there a way to add a managed folder as input of a python recipe in code?

Thanks,

Anna


Operating system used: Mac

Answers

  • AnnaZ
    AnnaZ Dataiku DSS Core Designer, Registered Posts: 3

    When I created a new managed folder in the python code, moved it to the current working zone, and then tried to use this new folder as input of another new python recipe, it showed a new error:

    code:

    new_folder = project.create_managed_folder("test")

    new_folder.move_to_zone("vdVhYpH")

    builder.with_input(new_folder)

    error:

    TypeError: Object of type DSSManagedFolder is not JSON serializable

  • AnnaZ
    AnnaZ Dataiku DSS Core Designer, Registered Posts: 3

    I found the solution....

  • CoreyS
    CoreyS Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 1,150 ✭✭✭✭✭✭✭✭✭

    Hey, @AnnaZ
    that's great! Would it be possible to share what your solution is? That way if someone else has a similar issue in the future they would be able to utilize your solution for their own benefit. Thanks!

  • pnw_ds
    pnw_ds Registered Posts: 4 ✭✭✭

    Same problem! Wish the solution was shared.

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,024 Neuron

    Please start a new thread, post your code and the error you get and we will help you.

  • pnw_ds
    pnw_ds Registered Posts: 4 ✭✭✭

    No need, I just had to make sure the dataset name in the input/output tab matched what was in my python recipe.

  • sanshekh
    sanshekh Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 2

    @pnw_ds
    Can you please share the code snippet as I've a similar issue

  • sanshekh
    sanshekh Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 2

    Hi All,

    I'm trying to export my final df of a python recipe to a managed folder & I get the below mentioned error:

    Job failed: Error in Python process: At line 35: <class 'Exception'>: Managed folder EyABfYFjV cannot be used : declare it as input or output of your recipe

    Any code snippet via which I can push my df data into a csv file & then when I go back to the recipes it should show the managed folder icon wherein my file should be present.

    Dataiku_.PNG

Setup Info
    Tags
      Help me…