Write in specific database from python recipe

Options
Florent1
Florent1 Registered Posts: 4 ✭✭✭

Hello,

Does someone have an example of a python recipe for writing a df into a specific database ?

Thank you very much !

Florent

Best Answer

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    edited 3:50PM Answer ✓
    Options

    Hi, That seems like a sensible approach, If you defined your project/global variable. This will result in multiple datasets being visible in the flow you can move them to a specific flow zone and even remove and not drop the contents depending on if you need to reuse these in the same flow.

    You can use something like his within a Scenario Python step. This won't work in an actual recipe but would work in scenario or notebook.

    import dataiku
    import pandas as pd, numpy as np
    from dataiku import pandasutils as pdu
    
    client = dataiku.api_client()
    project = client.get_default_project()
    
    # Read recipe inputs
    my_input_ds = dataiku.Dataset("year")
    my_dataframe = my_input_ds.get_dataframe()
    
    ts_var = dataiku.get_custom_variables()["timestamp_sql_table"]
    dataset_name = "my_dataset" + "_" + ts_var
    
    connection_name = "Postgres-Localhost"
    
    builder = project.new_managed_dataset(dataset_name)
    builder.with_store_into(connection_name)
    dataset = builder.create()
    
    #
    write_ds = dataiku.Dataset(dataset_name)
    write_ds.write_with_schema(my_dataframe)
    

    You can also create unmanaged SQL datasets using https://doc.dataiku.com/dss/latest/python-api/datasets-other.html#sql-dataset-programmatic-creation

    Let me know if that helps!

Answers

Setup Info
    Tags
      Help me…