Automating Flow
Hi
I have recently finished creating a flow that ingests data from a database, then it is run through an r recipe and then a sync recipe to upload to an S3 bucket.
I am now looking to automate this process. Is it possible to set up a scenario that runs through the flow and then upload the resulting dataset to a S3 bucket?
Answers
-
Manuel Alpha Tester, Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Dataiku DSS Adv Designer, Registered Posts: 193 ✭✭✭✭✭✭✭OptionsHi,
Yes, the functionality to automate a flow is called scenarios:
I hope this helps.
-
Hi,
Thanks for sharing these resources. I have set up a scenario and a trigger that runs every 5 minutes to build my datasets and run my recipes. I get a message saying the scenario was successful but my data is not updated in S3? Do you know why this would be? Because my datasets did not change in the 5 minutes? I set the trigger to go every 5 minutes to verify that it was working.
Thanks
-
Manuel Alpha Tester, Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Dataiku DSS Adv Designer, Registered Posts: 193 ✭✭✭✭✭✭✭Options
Hi,
Yes, if the input data does not change and the flow is the same, then the output should be the same.
If you want to be certain, clear the data from your output dataset and wait for the scenario to run again.
I hope this helps.