Reading folder contents from a managed folder

vishet
vishet Registered Posts: 17 ✭✭✭✭

Hi,

I am trying to write a recipe that reads from a managed folder and then creates a post request to an API with a file residing in the managed folder. The partial code for the recipe is as follows:

-------------------------------------------------------------------------

handle = dataiku.Folder("Ckfy9sf3")
paths = handle.list_paths_in_partition()
filenames= [path.replace("/","") for path in paths]
for path in filenames:
with handle.get_download_stream(path) as f:
headers = {
'accept': 'application/json',
'Authorization': bearer ,
}
files = {'file':(path, f.read())}
response = requests.post('URL of the API', headers=headers, files=files)
print(response.json())

----------------------------------------------------------------------

I am facing a problem where the API does not recognise the file like it should. I tried to perform this operation on my local machine with python(without managed folders of course and it worked just fine .My main question is: are files handled differently in dataiku and if yes, what are the conversions that I have to make to get my code to work?

Best Answer

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker
    edited July 17 Answer ✓

    Hi,

    the result of get_download_stream() can be passed as-is to the requests library. For example this works fine:

    import dataiku
    import os, requests
    
    handle = dataiku.Folder("a")
    paths = handle.list_paths_in_partition()
    filenames = [os.path.basename(path) for path in paths]
    for path in filenames:
        with handle.get_download_stream(path) as f:
            files = {'file':(path, f)}
            response = requests.post('https://httpbin.org/post', files=files)
            print(response)

    What is the error you see?

Setup Info
    Tags
      Help me…