Understanding if Python code is running in a recipe, scenario, web application, etc

info-rchitect

Hi,

We have made our own Dataiku Python package `dku` to standardize common code sequences into functions.

A simple example:

from dku import dku
from sf import sf
from dataiku.core.sql import SQLExecutor2

vars = dku.custom_variables

db = vars['snowflake_db']

schema = vars['snowflake_schema']

source_table = f"{db}.{schema}.mysourcetable"

with sf.sql('agg') as s:
    s.keep_remote = True
    s.text = f"""select part_name, count(distinct id) as id_cnt
from {source_table}
group by part_name"""
    s.submit()

post_write_statements = [
  f"grant select on table {db}.{schema}.myoutputtable to role ROLE1;",
  f"grant select on table {db}.{schema}.myoutputtable to role ROLE2;"
]

dku.set_post_write_statements('myoutputtable',
                              statements=post_write_statements)

# Write recipe outputs
myoutputtable= dataiku.Dataset("myoutputtable")
SQLExecutor2.exec_recipe_fragment(output_dataset=myoutputtable,
                              query=f"select * from {sf.sqls('agg').result_table_path}",
                              overwrite_output_schema=True)

Is there a way to detect if the code is running in a Python recipe, in a scenario, in a notebook, etc. I am trying to make the module intelligent enough to so it can expose/hide code functions as appropriate.

Eventually, the API should look something like this:

if in_recipe:
   ...
elif in_notebook:
   ...
etc

thx

---

Operating system used: Windows 10

Turribeach

Very interesting question, tagging myself. I suspect Dataiku knows but this might not be available to us users...

info-rchitect

@Turribeach
Really, this would be a great API to expose to users. I bet I can hack some methods by using the try-except pattern, but would prefer a dataiku approved solution.

import dataiku

if dataiku.in_python_recipe:
    ...
elif dataiku.in_notebook:

etc

HarizoR

Hi,

There is indeed no such thing directly exposed to the user. If you think that it can be a useful feature, feel free to write about it in the Product Ideas section.

Best,

Harizo

info-rchitect

@hari
@Turribeach
Expose an API to users that tells them where Python code is being executed - Dataiku Community