Auto trigger based on Dataset change
Hi all,
I have enabled an Auto Trigger for a scenario. I have used trigger based on 'Dataset modified' as seen in the picture below.
But as you can see I have used folders instead of Datasets here.
The problem I am facing is that, this scenario is triggered every 300 seconds even when there is no change in the folder files.
Can you please tell me how to make sure that this scenario gets triggered only when there is changes done to the folder?
Thanks in Advance
Answers
-
Emma Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer Posts: 52 DataikerOptionsHey @Akshay1
,In your screenshot, the 300 seconds is indicating how often Dataiku will poll the folder for changes not how often it will actually run. When applied to a folder "Trigger on dataset change" in Scenarios will trigger when a file is uploaded, modified, or deleted so it may be activating on instances that you would not expect.
A workaround could be to create a project variable reflecting some information related to the files within the folder and change when that changes (file count, build time, etc).
Hope that helps,
Emma
-
Hi @Emma
,Thanks for the response
By 'toll' you mean that the criteria for trigger is checked every 300 seconds right?
If yes, then I think I am on the same track already.
My current flow in Dataiku is:
1) Every 300 seconds I get the message on browser that 'Scenario started'. (My understanding is that this is when my criteria for trigger gets checked).
2) Even though no file in the folders are modified or changed or deleted. This step is immediately followed by 'Building of Datasets' that I have listed in the 'build/train' step in the scenario.
I don't understand why Building of the Datasets happen even when the criteria has failed?
Please let me know if I am missing something here.
Thanks once again!!!
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,757 NeuronOptionsHi, I use Dataset modified trigger on folders and I do not get executions when the files in the folders have not been changed. So while you assure us that "no file in the folders are modified or changed or deleted" I believe that you are not correct in that statement. Since you have 7 folders it will be hard to see which one is the one triggering the change, so I suggest you create a dummy scenario and test with each of the 7 folder to see which are the ones having changes. Finally please make sure you are not using any of these folders as an output in your flow as you will otherwise be writing to them and hence making a change while your trigger runs which will effectively make your trigger run again.