How to (if so) access Project Level Variables from (custom) python recipe?

UserBird

Simple question / use case:

1. I want to to set project level tokens for data connections to per project databases.
2. I understand, that there exists projects variables you can declare in json format.
3. I don't know how the heck access those variables from within a python recipe.
4. I need help, if this is something that's possible.

Disclaimer: I searched the docs up and down and only found this: https://doc.dataiku.com/dss/latest/advanced/variables_expansion.html#python but it doesn't work as described in the docs. I also searched the source code up and down, but couldn't make sense of it. I'm on 4.0.8

Alex_Combessie

Hello,

We would like to understand better what you want to achieve on 1.

Do you want to have multiple projects with similar flows on the same connection, and avoid having users on different projects write to the same tables? If so, you can have a look at https://doc.dataiku.com/dss/latest/connecting/relocation.html. The simplest way would be to add a ${projectKey}_ prefix in the connection menu, for the table name (and Hive name too on HDFS).

Is that what you have in mind?

It is possible to call custom project variables from Python code, but there could be other ways to do it depending on your use case. As for things that do not work, you could please share the code snippets and the error message?

Cheers,

Alex

UserBird

Ok, I should have elaborated on the context in the first place, sry...

I use Dataiku for ETL projects (ERP migration). I have developed a connector who does all the heavy lifting to the target instance. Now one project in dataiku is one migration project in real life.

Let's say I have tables A, B & C to upload to the target instance with receipte compute_A, compute_B & compute_C, then I only want to set the connection parameters (url, token, etc) once and that every recipe can access those instance parameters.

When everything is OK, I would just switch the staging parameters for production parameters in one single place.

UserBird

I must have been very tired, when I wrote this post, because now, doing:
`dataiku.get_custom_variables()` results as expected... Wow!
Thanks anyway!

Alex_Combessie

Great news! From the context of your project, it seems indeed that project variables should do the trick.

Note that if you want to create a complete industrialization workflow, you may be interested in the Automation node as well. In this case, we created a feature called Connection Remapping to remap connections from "dev" to "prod": https://doc.dataiku.com/dss/latest/bundles/importing.html#connection-remapping

Gustavo_Brian

Sorry not working for me in a "plugin":

self.customVariables = dataiku.get_custom_variables()
[2018/09/05-17:32:47.540] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils] - File "/disk2/dataiku-dss-4.2.2/python/dataiku/__init__.py", line 77, in get_custom_variables
[2018/09/05-17:32:47.547] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils] - "projectKey" : default_project_key() if project_key is None else project_key,
[2018/09/05-17:32:47.547] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils] - File "/disk2/dataiku-dss-4.2.2/python/dataiku/__init__.py", line 66, in default_project_key
[2018/09/05-17:32:47.547] [KNL-python-single-command-kernel-err-42113] [INFO] [dku.utils] - raise Exception("Please specify for which project key you want datasets list")

Alex_Combessie

This is working for me for a plugin recipe. What kind of plugin component are you developing? Can you give us more context regarding the goal of this plugin?

Gustavo_Brian

Thanks @Alex Combessie,

This is a custom "Dataset" component developed using the "Plugin developer" tools.
From within the python code I have:
self.customVariables = dataiku.get_custom_variables()

But it causes an error: "Please specify for which project key you want datasets list"

Alex_Combessie

Could you retry with dataiku.get_custom_variables(project_key=dataiku.default_project_key()) instead?

Gustavo_Brian

No luck

self.customVariables = dataiku.get_custom_variables(project_key=dataiku.default_project_key())
File "/disk2/dataiku-dss-4.2.2/python/dataiku/__init__.py", line 66, in default_project_key
raise Exception("Please specify for which project key you want datasets list")
Exception: Please specify for which project key you want datasets list