How to Get data from dataset to use it as Variables

Solved!
Richard_CDC
Level 2
How to Get data from dataset to use it as Variables

Hi,

I m using a scenario to update "Global variables" in my projet variables

I have a dataset With:

DIRAge
AURA35
PAUL48
.......

 

I want to put the column "DIR" in Variables to be able to used the data as filter in my dtaiku project. But it doesn t work because the result in my "Global Variables" have "[" and i can't use it. If I remove List() i have an error 

TypeError: DIR    AURA
Name: 0, dtype: object is not JSON serializable

 

My code

import dataiku

from dataiku.scenario import Scenario
scenario = Scenario()

df_data=dataiku.Dataset("Test_DIR").get_dataframe(infer_with_pandas=False)

df_data = df_data.iloc[0]

Scenario().set_project_variables(Var=list(df_data))

 

Result in "Project Variables"

Global variables

{
"Var": [
"AURA"
]
}

What can i use to replace list() ?


Operating system used: Windows


Operating system used: Windows

0 Kudos
1 Solution
arnaudde
Dataiker

If you just want the first element of the list you can get it by doing 

Scenario().set_project_variables(Var=list(df_data)[0])

View solution in original post

7 Replies
arnaudde
Dataiker

Hello,

Project variables should be valid json objects. In your example the variable contains an array so you need to escape the brackets ("[" should be "\["). You can use the python json library to encode python objects in json with the json.dumps function. 

Scenario().set_project_variables(Var=json.dumps(list(df_data)))


I hope this helps

0 Kudos
Richard_CDC
Level 2
Author

Thanks for the help, but now i have:

{
"Var": "[\"AURA\"]"
}

 

And i really don't know how to remove "[\ I would like to have as result: 

{
"Var": "AURA"
}

Any idea  ?

 

0 Kudos
arnaudde
Dataiker

If you just want the first element of the list you can get it by doing 

Scenario().set_project_variables(Var=list(df_data)[0])
Richard_CDC
Level 2
Author

Thank you it s working.

 

Now i want to build a dataset, but when i built de dataset 'Test_envoie_boucle', the filter just before doesn't apply.

2022-01-17_17h07_41.png

 I don't understand because i RECURSIVE_FORCED_BUILD the dataset. And when i do it manualy

that's working.

 Do i do something wrong in my code ?

 

import dataiku
import json
from dataiku.scenario import Scenario

scenario = Scenario()

df_data=dataiku.Dataset("Contact").get_dataframe(infer_with_pandas=False)
count = df_data.count()

df_data = df_data.iloc[3]

Scenario().set_project_variables(Var=list(df_data)[0])

Scenario().build_dataset('Test_envoie_boucle', buildt="RECURSIVE_FORCED_BUILD")

0 Kudos
arnaudde
Dataiker

The project variable `Var` probably does not hold the value you expect it to hold
Can you share the activity log from the jobs page so that we can investigate ? 

0 Kudos
akember182
Level 1

Hello I have a similar requirement.

I have a dataset (my_dataset) that looks like this

Code.                  Year

122.                     2021

I want to extract the 2021 figure and save it as a variable.

I know I can use code in a scenario along these lines 

import dataiku

from dataiku.scenario import Scenario
scenario = Scenario()

df_data=dataiku.Dataset("my_dataset").get_dataframe(infer_with_pandas=False)

df_data = df_data['Year']

Scenario().set_project_variables("year_variable"=list(df_data)[0])

 

But I can't quite seem to get it working, any advice?

0 Kudos

removed...

0 Kudos