Submit your inspiring success story or innovative use case to the 2022 Dataiku Frontrunner Awards! ENTER YOUR SUBMISSION

Auto trigger based on Dataset change

Akshay1
Level 2
Auto trigger based on Dataset change

Hi all,

I have enabled an Auto Trigger for a scenario. I have used trigger based on 'Dataset modified' as seen in the picture below.

4.JPG

But as you can see I have used folders instead of Datasets here. 

The problem I am facing is that, this scenario is triggered every 300 seconds even when there is no change in the folder files. 

Can you please tell me how to make sure that this scenario gets triggered only when there is changes done to the folder?

Thanks in Advance

0 Kudos
3 Replies
Emma
Dataiker
Dataiker

Hey @Akshay1 ,

In your screenshot, the 300 seconds is indicating how often Dataiku will poll the folder for changes not how often it will actually run. When applied to a folder "Trigger on dataset change" in Scenarios will trigger when a file is uploaded, modified, or deleted so it may be activating on instances that you would not expect. 

A workaround could be to create a project variable reflecting some information related to the files within the folder and change when that changes (file count, build time, etc). 

Hope that helps, 

Emma 

 

Akshay1
Level 2
Author

Hi @Emma

Thanks for the response 🙂

By 'toll' you mean that the criteria for trigger is checked every 300 seconds right?

If yes, then I think I am on the same track already.

My current flow in Dataiku is:

1) Every 300 seconds I get the message on browser that 'Scenario started'. (My understanding is that this is when my criteria for trigger gets checked). 

2) Even though no file in the folders are modified or changed or deleted. This step is immediately followed by 'Building of Datasets' that I have listed in the 'build/train' step in the scenario. 

I don't understand why Building of the Datasets happen even when the criteria has failed? 

Please let me know if I am missing something here.

Thanks once again!!!

 

 

 

0 Kudos
Turribeach
Level 4

Hi, I use Dataset modified trigger on folders and  I do not get executions when the files in the folders have not been changed. So while you assure us that "no file in the folders are modified or changed or deleted" I believe that you are not correct in that statement. Since you have 7 folders it will be hard to see which one is the one triggering the change, so I suggest you create a dummy scenario and test with each of the 7 folder to see which are the ones having changes. Finally please make sure you are not using any of these folders as an output in your flow as you will otherwise be writing to them and hence making a change while your trigger runs which will effectively make your trigger run again. 

0 Kudos