We're excited to announce that we're launching the second installment of Dataiku Product Days Register Now

Parameter file for DB connections

anthony
Level 2
Level 2
Parameter file for DB connections

Is there a way in DSS to take connection parameters for a database like host, user, password and all a from a parameter file.

0 Kudos
4 Replies
tim-wright
Neuron
Neuron

@anthony I'm not aware of any "clean" way to reference variables from within a parameter file that you provide. 

You can define "variables" in DSS (defined from within the project or from within the Admin tabs) which are stored and referenceable by other DSS elements or in other recipes. These are actually stored into a config file on the DSS server. Were you aware of this? Does it fail to meet your need in some way? (take a look around here: https://doc.dataiku.com/dss/latest/advanced/variables_expansion.html

Can you elaborate on how you were planning to use the parameter file? That might help the community best advise on how to accomplish something similar with DSS. 

Thanks,

Tim

 

 

 

 

anthony
Level 2
Level 2
Author

@tim-wright Thanks for the reply. My use case is to create a variable which can be updated on runtime so that the flow can process different set of data on each run. The data filter will be created using the variable. So is there a way where Dataiku accepts value for the variable on runtime other than manually updating the variable value in global variables before running the flow. Being new to Dataiku, is there any other way to implement this. Thanks in advance

0 Kudos
tgb417
Neuron
Neuron

@anthony ,

I was starting the new Advanced Designer Learning Path over at the Dataiku Academy. In the course Variables 101 there is a mention of updating DSS Variables in Scenarios and Python and R code through the REST API.  It sounded like these features might help in your use case.

However, I've not finished the course work on this part of the system.  So, I'm sorry I don't have a lot more on this possibility.  Hope that might help a little bit.

cc: @CoreyS 

--Tom
0 Kudos
tim-wright
Neuron
Neuron

@anthony- sorry for the delayed response. I was thinking more or less exactly what @tgb417 mentioned in his post. If you are comfortable with code recipes you can build python code that will update the parameters using the REST API after each run. This is helpful if you know what you want to update your variables to. The limitation here is that the account running the recipe (and updating the variable) will need sufficient privileges to do so.

Minimally viable Python example of how to update the variables using code. Note: Here, I am manually deciding what to update the variables to (hard-coded). You will likely need an automated way (code based) to determine that for you (for example: get value of the last variable and increment by 1). 

import dataiku
api = dataiku.api_client()      # API Client (will have permissions of the user who runs this code)

dss_variables = api.get_variables() # Scope = DSS (these are system variables across DSS, not the variables for the project)

# Get the current project
proj = api.get_default_project()      # Get the project within which we are running this code. Use api.get_project(<project_name>) method to use a different project
proj_variables = proj.get_variables() # Project variables

#{u'local': {u'local1': u'local1'}, u'standard': {u'global1': u'global1'}}   

new_proj_variables = variables.copy() # Create copy of the existing variables 

# Change the local variables
new_proj_variables['local']['local1'] = 'local2'          # Modify existing variable
new_proj_variables['local']['a_new_local_var'] = 14       # Add new variable
proj.set_variables(new_variables)                         # UPDATE the variables for this project

proj.get_variables()    # Fetch the New variables
# {u'local': {u'a_new_local_var': 14, u'local1': u'local2'}, u'standard': {u'global1': u'global1'}}

 

A banner prompting to get Dataiku DSS