Survey banner
The Dataiku Community is moving to a new home! New posts are now disabled and the community will shortly be in temporary read only mode: LEARN MORE

Understanding if Python code is running in a recipe, scenario, web application, etc

info-rchitect
Level 6
Understanding if Python code is running in a recipe, scenario, web application, etc

Hi,

We have made our own Dataiku Python package `dku` to standardize common code sequences into functions.

A simple example:

 

 

 

 

from dku import dku
from sf import sf
from dataiku.core.sql import SQLExecutor2

vars = dku.custom_variables

db = vars['snowflake_db']

schema = vars['snowflake_schema']

source_table = f"{db}.{schema}.mysourcetable"

with sf.sql('agg') as s:
    s.keep_remote = True
    s.text = f"""select part_name, count(distinct id) as id_cnt
from {source_table}
group by part_name"""
    s.submit()

post_write_statements = [
  f"grant select on table {db}.{schema}.myoutputtable to role ROLE1;",
  f"grant select on table {db}.{schema}.myoutputtable to role ROLE2;"
]

dku.set_post_write_statements('myoutputtable',
                              statements=post_write_statements)

# Write recipe outputs
myoutputtable= dataiku.Dataset("myoutputtable")
SQLExecutor2.exec_recipe_fragment(output_dataset=myoutputtable,
                              query=f"select * from {sf.sqls('agg').result_table_path}",
                              overwrite_output_schema=True)

 

 

 

 

Is there a way to detect if the code is running in a Python recipe, in a scenario, in a notebook, etc.  I am trying to make the module intelligent enough to so it can expose/hide code functions as appropriate.

Eventually, the API should look something like this:

 

 

 

 

 

if in_recipe:
   ...
elif in_notebook:
   ...
etc

 

 

 

 

 

thx


Operating system used: Windows 10

0 Kudos
4 Replies
Turribeach

Very interesting question, tagging myself. I suspect Dataiku knows but this might not be available to us users...

info-rchitect
Level 6
Author

@Turribeach Really, this would be a great API to expose to users.  I bet I can hack some methods by using the try-except pattern, but would prefer a dataiku approved solution.

 

 

 

import dataiku

if dataiku.in_python_recipe:
    ...
elif dataiku.in_notebook:

etc

 

 

0 Kudos
HarizoR
Developer Advocate

Hi,

There is indeed no such thing directly exposed to the user. If you think that it can be a useful feature, feel free to write about it in the Product Ideas section.

 

Best,

Harizo

0 Kudos