Write in specific database from python recipe

Florent1
Florent1 Registered Posts: 4 ✭✭✭

Hello,

Does someone have an example of a python recipe for writing a df into a specific database ?

Thank you very much !

Florent

Best Answer

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,225 Dataiker
    edited July 17 Answer ✓

    Hi, That seems like a sensible approach, If you defined your project/global variable. This will result in multiple datasets being visible in the flow you can move them to a specific flow zone and even remove and not drop the contents depending on if you need to reuse these in the same flow.

    You can use something like his within a Scenario Python step. This won't work in an actual recipe but would work in scenario or notebook.

    import dataiku
    import pandas as pd, numpy as np
    from dataiku import pandasutils as pdu
    
    client = dataiku.api_client()
    project = client.get_default_project()
    
    # Read recipe inputs
    my_input_ds = dataiku.Dataset("year")
    my_dataframe = my_input_ds.get_dataframe()
    
    ts_var = dataiku.get_custom_variables()["timestamp_sql_table"]
    dataset_name = "my_dataset" + "_" + ts_var
    
    connection_name = "Postgres-Localhost"
    
    builder = project.new_managed_dataset(dataset_name)
    builder.with_store_into(connection_name)
    dataset = builder.create()
    
    #
    write_ds = dataiku.Dataset(dataset_name)
    write_ds.write_with_schema(my_dataframe)
    

    You can also create unmanaged SQL datasets using https://doc.dataiku.com/dss/latest/python-api/datasets-other.html#sql-dataset-programmatic-creation

    Let me know if that helps!

Answers

Setup Info
    Tags
      Help me…