How to Get data from dataset to use it as Variables

Options
Richard_CDC
Richard_CDC Registered Posts: 6 ✭✭✭

Hi,

I m using a scenario to update "Global variables" in my projet variables

I have a dataset With:

DIRAge
AURA35
PAUL48
.......

I want to put the column "DIR" in Variables to be able to used the data as filter in my dtaiku project. But it doesn t work because the result in my "Global Variables" have "[" and i can't use it. If I remove List() i have an error

TypeError: DIR    AURA
Name: 0, dtype: object is not JSON serializable

My code

import dataiku

from dataiku.scenario import Scenario
scenario = Scenario()

df_data=dataiku.Dataset("Test_DIR").get_dataframe(infer_with_pandas=False)

df_data = df_data.iloc[0]

Scenario().set_project_variables(Var=list(df_data))

Result in "Project Variables"

Global variables

{
"Var": [
"AURA"
]
}

What can i use to replace list() ?


Operating system used: Windows


Operating system used: Windows

Best Answer

Answers

  • arnaudde
    arnaudde Dataiker Posts: 52 Dataiker
    Options

    Hello,

    Project variables should be valid json objects. In your example the variable contains an array so you need to escape the brackets ("[" should be "\["). You can use the python json library to encode python objects in json with the json.dumps function.

    Scenario().set_project_variables(Var=json.dumps(list(df_data)))


    I hope this helps

  • Richard_CDC
    Richard_CDC Registered Posts: 6 ✭✭✭
    Options

    Thanks for the help, but now i have:

    {
    "Var": "[\"AURA\"]"
    }

    And i really don't know how to remove "[\ I would like to have as result:

    {
    "Var": "AURA"
    }

    Any idea ?

  • Richard_CDC
    Richard_CDC Registered Posts: 6 ✭✭✭
    Options

    Thank you it s working.

    Now i want to build a dataset, but when i built de dataset 'Test_envoie_boucle', the filter just before doesn't apply.

    2022-01-17_17h07_41.png

     I don't understand because i RECURSIVE_FORCED_BUILD the dataset. And when i do it manualy

    that's working.

    Do i do something wrong in my code ?

    import dataiku
    import json
    from dataiku.scenario import Scenario

    scenario = Scenario()

    df_data=dataiku.Dataset("Contact").get_dataframe(infer_with_pandas=False)
    count = df_data.count()

    df_data = df_data.iloc[3]

    Scenario().set_project_variables(Var=list(df_data)[0])

    Scenario().build_dataset('Test_envoie_boucle', buildt="RECURSIVE_FORCED_BUILD")

  • arnaudde
    arnaudde Dataiker Posts: 52 Dataiker
    Options

    The project variable `Var` probably does not hold the value you expect it to hold
    Can you share the activity log from the jobs page so that we can investigate ?

  • akember182
    akember182 Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 2
    Options

    Hello I have a similar requirement.

    I have a dataset (my_dataset) that looks like this

    Code. Year

    122. 2021

    I want to extract the 2021 figure and save it as a variable.

    I know I can use code in a scenario along these lines

    import dataiku

    from dataiku.scenario import Scenario
    scenario = Scenario()

    df_data=dataiku.Dataset("my_dataset").get_dataframe(infer_with_pandas=False)

    df_data = df_data['Year']

    Scenario().set_project_variables("year_variable"=list(df_data)[0])

    But I can't quite seem to get it working, any advice?

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,708 Neuron
    Options
Setup Info
    Tags
      Help me…