How to automate the process and trigger at schedule time
Hi,
I am new to dataiku.
Currently I build the process which has excel file
Koolas input. The single excel file consist of 5 different sheets.
For the 1 st process it need to read 1 st sheet and prepare the recipe as per the steps configured.
I need to run these process flow in automatic process without manual upload.
Can you help me out how to make automatic process and the steps how to configure.
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 DataikerOptionsHi @SUSHIL
,To automate you can use build dataset steps in a scenario with the time-based trigger. You can use the plugin below to import sheets.
https://knowledge.dataiku.com/latest/courses/automation/scenarios-summary.html
If you want to avoid manually uploading the file the file would need to be published to some storage where Dataiku can read it from Cloud Storage, Sharepoint, etc.
Let me know if that helps.
-
Manuel Alpha Tester, Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Dataiku DSS Adv Designer, Registered Posts: 193 ✭✭✭✭✭✭✭OptionsHi,
The functionality to automate a flow in Dataiku is called Scenarios and it includes time-based triggers (amongst other types).
The challenge that you have is that your input data is an uploaded dataset. By definition, this dataset needs to be uploaded before you can re-run the flow. It is not a "live" data connection.
You have some alternatives to address this challenge:
- Change your input data to a data connection, such an SQL table. Then the flow can be triggered automatically and it will read the latest data from the table;
- Change to google sheets. We actually have a plugin that allows you to treat a Google Sheet as a data connection;
- Create a Dataiku Application. Whilst you would still need to provide the Excel file as input, it can be used to automate the subsequent flow.
I hope this helps.