-
Scenario - Building the Dataset (Sequence Logic)
I am creating a scenario for a workflow. I need to understand the logic of building the dataset in terms of the datasets. So, I have the following sequence of the datasets in the flow A->B->C->. If I include all the datasets in one build step, would the DSS adhere to the sequence or do I need to create separate build step…
-
using scenarioTriggerName in a scenario step condition
Hi, how can I reference the trigger name in a step condition of a scenario. In my situation I want to run step 1 only if the scenario has not been triggered manually, that means triggerName != 'Manual trigger'. But the variable 'triggerName' does not return any value. Thank you, Romana
-
Application-as-recipes and SQL scripts reusable for different connections
While turning a flow of SQL script recipes into an application-as-recipe we encountered some difficulties making it generalisable for new input SQL tables that have a different connection/schema/tablename as the original input table. We are interested to learn how other users dealt with this and what best practices are…
-
Scenario run Last Business date of the month
Dear community, I have just started using Dataiku and creating a couple of data models. I have used the automation scenario with simple by default settings offered by Dataiku. But now I would like to automate the run for one of my data models to run on a monthly basis for the Last Business date of the month. Anyone on how…
-
automatic updating of data with the scenario
Hi, I use an api to get new data every day. I would like to stack those data on a unique dataset. I have made this flow, but I can't build it recursively as it is a cycle and will produce stack overflow. I think my solution is a bit tricky, is there a more straightforward way to do it? - My flow first get data from api,…
-
Run a complexe SQL "script" to create tables in Bigquery (GCP)
Hello everyone, I have a serious trouble when i want to process my data from one table in bigquery to another via the execution of a sql script. See the error below. (This step writen in sql is a processing step which should run in bigquery compute instance) Do you have any idea? Thanks in advandfor your answer
-
Scenario triggered by an external change
I am trying to create a scenario that is triggered by an external change to a dataset. Specifically, I have dataset created from a Redshift table. And I have a recipe that processes this dataset. I want to be able to trigger the recipe every time a change is made to the Redshift table. Any change to that table will be done…
-
Scenario SQL query change trigger
I already posted this question earlier , but perhaps it wasn't clear, so I'll try to be more precise here. Custom python trigger code uses t.fire() to trigger the scenario when the condition is met. What is the equivalent of that t.fire() in the SQL query change trigger? Thank you
-
Data quality : Monitoring on datasets processing
Hi, I'm asking about how DSS monitors issues during datasets processing. I see two kinds of potential issues: * Volume : Inconsistant number of records in a dataset (eg : I expected at least 1k records per day for my "webtraffic" dataset) * Schema / values: One or more rows have fields that don't respect the defined schema…
-
Exporting JupyterNotebook to email using Scenario Steps "Send Message"
I am trying to export Jupyter Notebook and want to send it to email using DataIku Scenarios by adding Step “Send Message” using outlook mail client. Within Jupyter Notebook, I have a code that hides all code cells and only displays each code cell output when I want to export the Notebook. However, when I am trying to send…
