Using Dataiku

Project variables are not a good place to store run time data. Scenario variables fit that use case much better. However I feel your post doesn't give enough information of what exactly you are trying to achieve and seems to be trying to reinvent the wheel. Why are you recording start and end times of some partial scenario steps? This information is already stored by Dataiku and available both from Dataiku API and in the Internal Stats dataset.

Project variables are not a good place to store run time data. Scenario variables fit that use case much better. However I feel your post doesn't give enough information of what exactly you are trying to achieve and seems to be trying to reinvent the wheel. Why are you recording start and end times of some partial scenario steps? This information is already stored by Dataiku and available both from Dataiku API and in the Internal Stats dataset.

in data warehouse, you make a diagram that data will be fetched from different sources. all sources can be shown.

Exactly and since Dataiku can connect to any possible data source available in most companies it's imposible to make a data architecture diagram without knowing all your sources but also all your outputs. I think what you want is a general Dataiku Architecture diagram which your Dataiku Account Manager/Customer Success Manager should be able to provide.

Yeah, sure! Here goes:

There are various ways to do that. The easiest way is to use a Prepare Recipe and add a step, Transpose.

https://doc.dataiku.com/dss/11/preparation/processors/transpose.html

Let us know if that doesn't meet your requirements.

Hi @kai_fang,

pandas.HDFStore requires a local file path, so in order to load if from a managed folder, we first need to copy it from the folder to a temporary local file.

For example:

import os

import shutil

import tempfile

import dataiku

import pandas as pd

folder = dataiku.Folder("MY_FOLDER")

with tempfile.TemporaryDirectory() as temp_dir:

    temp_path = os.path.join(temp_dir, "temp.h5")

    # Copy the file from the remote folder to a temp local file
    with folder.get_download_stream("FILE_IN_FOLDER.h5") as folder_stream:
        with open(temp_path, "wb") as temp_stream:
            shutil.copyfileobj(folder_stream, temp_stream)

    # Load the temp HDF file
    with pd.HDFStore(temp_path, mode="r") as hdf:
        hdf_keys = hdf.keys()

I wasn't aware of this plugin and it is pretty cool that this can be supported in such way but in my view this makes the Administration of the security model a bigger overhead. Since the original poster is trying to prevent "unintended data mishandling" my approach would be to monitor projects using the Dataiku Python API which can easily detect which connections are being used in each project.

Hi, there are no built-in features for this but you can search inside recipe code using the Dataiku Python API:

import dataiku

client_handle = dataiku.api_client()
variables_to_search = ['var1', 'var2']
project_handle = client_handle.get_project('some project key')
python_recipes = [i for i in project_handle.list_recipes() if i['type'] in ['python']]
for python_recipe in python_recipes:
  recipe_name = python_recipe['name']
  recipe_handle = project_handle.get_recipe(recipe_name)
  recipe_script = recipe_handle.get_settings().get_payload().lower()
  if recipe_script:
    for var in variables_to_search:
      if var.lower() in recipe_script:
        print(f'Found variable {var} in recipe {recipe_name}')

It shouldn't be too hard to customise this code to search for secrets. The following Python packages should help:

https://pypi.org/project/ggshield/

https://pypi.org/project/whispers/