How to load a file to Resources directory and have available at run time

clayms
clayms Registered Posts: 52 ✭✭✭✭

How can I simply upload a zipped certificate file to the resources directory and then update the code environment and have that certificate file available at run time?

I upload the file, change the script to include a Environment Variable to point to the folder, and give permissions to the folder. Then at run time, the environment variable does contain the path to the runtime Resources folder, but there is nothing in the resources folder.

Where is a documented example of how to upload a file to the Code Environment Resources Directory and make it available at runtime?


Operating system used: centos

Tagged:

Answers

  • Zach
    Zach Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 153 Dataiker
    edited July 17

    Hi @clayms
    ,

    I'm able to access a file that I uploaded to the resources directory of a code env, as shown below:

    CB927DEE-DF6A-4141-894E-325D76D1441D.png

    Initialization script:

    ## Base imports
    from dataiku.code_env_resources import clear_all_env_vars
    from dataiku.code_env_resources import set_env_path
    from dataiku.code_env_resources import set_env_var
    
    # Clears all environment variables defined by previously run script
    clear_all_env_vars()
    
    # Set the path to the cert file
    set_env_path("MY_CERT_PATH", "cert/cert.crt")

    I can then access the cert using the following code in a notebook with the kernel set to my code env:

    import os
    
    cert_path = os.environ["MY_CERT_PATH"]
    print("cert_path:", cert_path)
    
    with open(cert_path, "r") as file:
        print(file.read())

    Could you please double check that the path to the file is correct? For example, in my above test, the cert is located in a subdirectory called "cert", so I set the relative path to "cert/cert.crt".

    Please also make sure that you've updated the code env after changing the init script and uploading the file. If you're trying to access the cert from a notebook, please also make sure that you've restarted the kernel after updating the code env.

    Additionally, per our documentation, please make sure that your code env meets the following requirements:

    • Code environment resources require the core packages to be installed.
    • Code environment resources are not supported on external conda code environments.
    • Code environment resources are not made available to Spark executors.

    If you're still not able to get it working, please provide provide the following information:

    • A screenshot of the Resources tab of your code env similar to my screenshot above.
    • The code that you're using to access the cert file, as well as the error message you're getting.
    • How you're accessing the resources script. For example, are you accessing it from a Python recipe, or a notebook?
    • Are you using containerized execution, or is your Python code running locally?

    Thanks,

    Zach

  • clayms
    clayms Registered Posts: 52 ✭✭✭✭

    See my answer to a question posted 12 or so hours after mine here :

    https://community.dataiku.com/t5/Setup-Configuration/How-to-add-a-file-to-the-Resources-directory-so-that-it-is/m-p/32917/highlight/true#M2938

    The documentation is poor. Also, I was told by Dataiku to check the "Rebuild env", then rebuild. This was the issue. That said, I did have to rebuild the environment for the file to be available in a container. I already had the "copy" option selected in the containerized so that was not the issue.

    The interplay between the "rebuild env" and "Run resources init script" and the options in Code Env > Containerized Execution > Resources initialization and when files get downloaded, when what you uploaded gets removed, and when those files gets put into a container and poorly documented and should be revised.

  • saulleon
    saulleon Registered Posts: 22 ✭✭✭✭

    Code-Env for containers is missing the path

    ENV points me to this location:

    enviroment01.png

    But files are located in this other location:

    enviroment02.png

Setup Info
    Tags
      Help me…