Rebuild/Refresh Snowflake Database in Scenario
Hi
I use Dataiku with Snowflake.
My Project Flow is the same as the picture below. (Sample)
Next is Scenario Python Script.
When Scenario works, I want to refresh/rebuild 'TRAFFIC' data in Snowflake Database.
How do I make Script??
Thank you.
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 DataikerOptionsHi @DaeWangIma
,
Since the traffic dataset is a input dataset this is "refreshed" everytime you read it again from a dependent recipe.What you can do here is do a recursive build via the scenario.
You can start the scenario based on SQL trigger for example to identify if the underlying data changed then re-run the scenario
https://doc.dataiku.com/dss/latest/scenarios/triggers.html#sql-triggersYou don't need to include custom python step a simple dataset build recursively should do the trick of any of the dependent datasets in the right of your flow.
https://knowledge.dataiku.com/latest/data-preparation/pipelines/concept-dataset-build-modes.html
Hope that helps!