Parameter file for DB connections
Is there a way in DSS to take connection parameters for a database like host, user, password and all a from a parameter file.
Answers
-
tim-wright Partner, L2 Designer, Snowflake Advanced, Neuron 2020, Registered, Neuron 2021, Neuron 2022 Posts: 77 Partner
@anthony
I'm not aware of any "clean" way to reference variables from within a parameter file that you provide.You can define "variables" in DSS (defined from within the project or from within the Admin tabs) which are stored and referenceable by other DSS elements or in other recipes. These are actually stored into a config file on the DSS server. Were you aware of this? Does it fail to meet your need in some way? (take a look around here: https://doc.dataiku.com/dss/latest/advanced/variables_expansion.html)
Can you elaborate on how you were planning to use the parameter file? That might help the community best advise on how to accomplish something similar with DSS.
Thanks,
Tim
-
@tim-wright
Thanks for the reply. My use case is to create a variable which can be updated on runtime so that the flow can process different set of data on each run. The data filter will be created using the variable. So is there a way where Dataiku accepts value for the variable on runtime other than manually updating the variable value in global variables before running the flow. Being new to Dataiku, is there any other way to implement this. Thanks in advance -
tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron
@anthony
,I was starting the new Advanced Designer Learning Path over at the Dataiku Academy. In the course Variables 101 there is a mention of updating DSS Variables in Scenarios and Python and R code through the REST API. It sounded like these features might help in your use case.
However, I've not finished the course work on this part of the system. So, I'm sorry I don't have a lot more on this possibility. Hope that might help a little bit.
cc: @CoreyS
-
tim-wright Partner, L2 Designer, Snowflake Advanced, Neuron 2020, Registered, Neuron 2021, Neuron 2022 Posts: 77 Partner
@anthony
- sorry for the delayed response. I was thinking more or less exactly what @tgb417
mentioned in his post. If you are comfortable with code recipes you can build python code that will update the parameters using the REST API after each run. This is helpful if you know what you want to update your variables to. The limitation here is that the account running the recipe (and updating the variable) will need sufficient privileges to do so.Minimally viable Python example of how to update the variables using code. Note: Here, I am manually deciding what to update the variables to (hard-coded). You will likely need an automated way (code based) to determine that for you (for example: get value of the last variable and increment by 1).
import dataiku api = dataiku.api_client() # API Client (will have permissions of the user who runs this code) dss_variables = api.get_variables() # Scope = DSS (these are system variables across DSS, not the variables for the project) # Get the current project proj = api.get_default_project() # Get the project within which we are running this code. Use api.get_project(<project_name>) method to use a different project proj_variables = proj.get_variables() # Project variables #{u'local': {u'local1': u'local1'}, u'standard': {u'global1': u'global1'}} new_proj_variables = variables.copy() # Create copy of the existing variables # Change the local variables new_proj_variables['local']['local1'] = 'local2' # Modify existing variable new_proj_variables['local']['a_new_local_var'] = 14 # Add new variable proj.set_variables(new_variables) # UPDATE the variables for this project proj.get_variables() # Fetch the New variables # {u'local': {u'a_new_local_var': 14, u'local1': u'local2'}, u'standard': {u'global1': u'global1'}}