Writing files to a folder in a project

Options
info-rchitect
info-rchitect Registered Posts: 169 ✭✭✭✭✭✭

Hi,

I manually created a folder in my project and want to download a zip file, via REST API, into it. I get this permissions error:

Permission denied: '/data/dataiku/data_dir/managed_folders/MYPROJECT/pMlFp07H/blah_2023-08-30T12-28-51_kdfs.zip

I know how to write to a folder with the folder handle. Is there anyway I can workaround this so I can download directly from a requests call?


Operating system used: Windows 10

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,737 Neuron
    Options

    Can you please post your code?

  • info-rchitect
    info-rchitect Registered Posts: 169 ✭✭✭✭✭✭
    edited July 17
    Options
                    resp = requests.get(zip_url, stream=True)
                    if resp.status_code == 200:
                        total = int(resp.headers.get("content-length", 0))
                        file_path = (
                            file_path if file_path else self.controller._cache_dir / f"{zipname}.zip"
                        )
                        with open(file_path, "wb") as file, tqdm(
                            desc=f"Downloading to {file_path}",
                            total=total,
                            unit="B",
                            unit_scale=True,
                            unit_divisor=1024,
                        ) as bar:
                            for data in resp.iter_content(chunk_size=1024):
                                size = file.write(data)
                                bar.update(size)
  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,737 Neuron
    Options

    I am bit confused on what you are trying to do. First you say:


    @info-rchitect
    wrote:
    I manually created a folder in my project and want to download a zip file, via REST API, into it. I get this permissions error:

    "download a zip file into it" sure means to upload a file to the folder not download. But then you say:


    @info-rchitect
    wrote:
    I know how to write to a folder with the folder handle. Is there anyway I can workaround this so I can download directly from a requests call?

    "Write to a folder" indicates you are uploading to a folder but then you say "so I can download directly from a requests call". So are you trying put (upload) or get (download) a file from managed folder?

    Now to your code. The code you posted is pure Python code trying to write to a folder. I see nowhere you using the relevant REST API call: https://doc.dataiku.com/dss/api/12/rest/#managed-folders-managed-folder-contents-post

    You get permission denied with your code because Dataiku will not allow you to execute code that writes directly to OS folders otherwise imagine what a rogue user could do. So why can't you use the Dataiku Python APIs to upload your file?

  • info-rchitect
    info-rchitect Registered Posts: 169 ✭✭✭✭✭✭
    Options

    I was unaware of the POST REST APIs, thx.

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,737 Neuron
    Options

    No worries. It is not a well known API but it’s basically the backend of the dataikuapi package which wraps around the REST API.

Setup Info
    Tags
      Help me…