How do I read a yml file from the dataiku library?

pnkj

I have a yml config file and want to read it in my notebooks, how do I do that?

Turribeach

https://python.land/data-processing/python-yaml

pnkj

https://community.dataiku.com/discussion/comment/43903#Comment_43903

I am not able to find a path to the file. It is showing error

Turribeach

You are using a relative path without even known what's your current directory. Also Dataiku does not allow you to read files from arbitrary directories, you should use a Dataiku managed folder.

Ignacio_Toledo

https://community.dataiku.com/discussion/comment/43904#Comment_43904

We had the same kind of use case, as we maintained some configuration yaml files in a bitbucket repository, so it seemed a good idea to connect it to the Project's "Libraries".

To find the path of your file I would use:

import os

pythonpaths = os.environ['PYTHONPATH'].split(':')


for pp in pythonpaths:

    local_lib_path = re.findall('.*lib/python$', pp)

    if local_lib_path :

        path = local_lib_path [0] + '/'


yaml_file = f'{path}/insights_config.yml'

This should work, but it is not bullet proof, and there are special case that might need to be taken care of.

I think @Turribeach solution might be more robust. But I also appreciate the convenience of being able to edit the yaml file from the Dataiku editor.

Ignacio_Toledo

Hi @pnkj!

I found a better and robust solution. I'm surprised I didn't find it before:

import dataiku

client = dataiku.api_client()
project = client.get_default_project()
library = project.get_library()

with library.get_file('insights_config.yml') as fin:
    data = yaml.safe_load(fin)

More information here:
Project libraries - Dataiku Developer Guide

Turribeach

https://community.dataiku.com/discussion/comment/44533#Comment_44533

This is a very good option IF you want to have the YAML file stored as a project library. There is even an API to add a file to the project library so you could do this programmatically. Furthermore there are now even APIs to add a project library from Git programmatically and even push changes to the remote Git.

Having said that this may not be suitable for all use cases. For instance if the YAML file is holding data then the project library is not a good place for it and a managed folder is a better choice. Also if the YAML needs to be placed in a network share or cloud bucket then again a managed folder will be a better choice.