Add a dataset from a data collection in Python

Jean-Philippe

Hi,

Is it possible in Dataiku to import a dataset from a Data Collection into a project, using the Python API ? (I mean from a recipe in the target project, not in the source project).

I am looking for a way to do this via a Python recipe or through the Dataiku Python API client, without having to do it manually through the interface (Add Item→ Data Catalog→ … → Use)

Thank you

FlorentD

Hi,

You need to have permissions on both projects. Then something like this

client = dataiku.api_client()
collection.get_data_collection('<YOUR COLLECTION ID>')
// Then you have to find the object you want to use
// get the type, project it belong to, and its name
// suppose it is the first item of the collection
item = collection.list_objects("dict)[0]
// Access to the project from
project_from =  client.get_project(item.get('projectKey'))
setttings = project_from.get_settings()
settings.add_exposed_object(item.get('type'), item.get('id'), '<DESTINATION_PROJECT_KEY>')
settings.save()

This should do the trick.

Have a good day.

Jean-Philippe

Hi,

Thank you for your feedback. Indeed, it's only works if the user has rights on the source project. I tested with a colleague and only enabling the "Admin" right on the source project allows importing the dataset from the data catalog. This bothers me a lot because through the interface, a user can import a dataset from the data catalog without having any rights on the source project. I found an alternative by importing through a database connection, but I'm less satisfied with it :

import_definition = project.init_tables_import()
import_definition.add_sql_table("my_sql_connection", "schema_of_the_databse", "name_of_the_table")

prepared_import = import_definition.prepare()
future = prepared_import.execute()

import_result = future.wait_for_result()

Turribeach

While clearly not fully automated are you aware you can bulk import datasets using the Database Explorer? (Add Item ⇒ Database Explorer).

Jean-Philippe

Hi,

Thanks for this tip, I wasn't aware of it! It would definitely be great to see similar bulk import functionality extended to Data Collections in the future to avoid having to import tables one by one, even if it's not fully automated.