Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi
I use Dataiku with Snowflake.
My Project Flow is the same as the picture below. (Sample)
Next is Scenario Python Script.
When Scenario works, I want to refresh/rebuild 'TRAFFIC' data in Snowflake Database.
How do I make Script??
Thank you.
Hi @DaeWangIma ,
Since the traffic dataset is a input dataset this is "refreshed" everytime you read it again from a dependent recipe.
What you can do here is do a recursive build via the scenario.
You can start the scenario based on SQL trigger for example to identify if the underlying data changed then re-run the scenario
https://doc.dataiku.com/dss/latest/scenarios/triggers.html#sql-triggers
You don't need to include custom python step a simple dataset build recursively should do the trick of any of the dependent datasets in the right of your flow.
https://knowledge.dataiku.com/latest/data-preparation/pipelines/concept-dataset-build-modes.html
Hope that helps!