How to automate the process and trigger at schedule time

SUSHIL
Level 3
How to automate the process and trigger at schedule time

Hi,  

I am new to dataiku. 

Currently I build the process which has  excel file

Koolas input. The  single excel file consist of 5 different sheets. 

For the 1 st process it need to read 1 st sheet and  prepare the recipe as per the steps configured. 

I need to run these process flow in automatic process without manual upload. 

Can you help me out how to make automatic process and the  steps how to configure. 

0 Kudos
2 Replies
AlexT
Dataiker

Hi @SUSHIL ,

To automate you can use build dataset steps in a scenario with the time-based trigger. You can use the plugin below to import sheets

https://knowledge.dataiku.com/latest/courses/automation/scenarios-summary.html 

If you want to avoid manually uploading the file the file would need to be published to some storage where Dataiku can read it from Cloud Storage,  Sharepoint, etc.

Let me know if that helps.

0 Kudos
Manuel
Dataiker Alumni

Hi,

The functionality to automate a flow in Dataiku is called Scenarios and it includes time-based triggers (amongst other types).

The challenge that you have is that your input data is an uploaded dataset. By definition, this dataset needs to be uploaded before you can re-run the flow. It is not a "live" data connection.

You have some alternatives to address this challenge:

  • Change your input data to a data connection, such an SQL table. Then the flow can be triggered automatically and it will read the latest data from the table;
  • Change to google sheets. We actually have a plugin that allows you to treat a Google Sheet as a data connection;
  • Create a Dataiku Application. Whilst you would still need to provide the Excel file as input, it can be used to automate the subsequent flow.

I hope this helps.

0 Kudos