Auto trigger based on Dataset change

Options
Akshay1
Akshay1 Registered Posts: 4 ✭✭✭

Hi all,

I have enabled an Auto Trigger for a scenario. I have used trigger based on 'Dataset modified' as seen in the picture below.

4.JPG

But as you can see I have used folders instead of Datasets here.

The problem I am facing is that, this scenario is triggered every 300 seconds even when there is no change in the folder files.

Can you please tell me how to make sure that this scenario gets triggered only when there is changes done to the folder?

Thanks in Advance

Answers

  • Emma
    Emma Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer Posts: 52 Dataiker
    Options

    Hey @Akshay1
    ,

    In your screenshot, the 300 seconds is indicating how often Dataiku will poll the folder for changes not how often it will actually run. When applied to a folder "Trigger on dataset change" in Scenarios will trigger when a file is uploaded, modified, or deleted so it may be activating on instances that you would not expect.

    A workaround could be to create a project variable reflecting some information related to the files within the folder and change when that changes (file count, build time, etc).

    Hope that helps,

    Emma

  • Akshay1
    Akshay1 Registered Posts: 4 ✭✭✭
    Options

    Hi @Emma
    ,

    Thanks for the response

    By 'toll' you mean that the criteria for trigger is checked every 300 seconds right?

    If yes, then I think I am on the same track already.

    My current flow in Dataiku is:

    1) Every 300 seconds I get the message on browser that 'Scenario started'. (My understanding is that this is when my criteria for trigger gets checked).

    2) Even though no file in the folders are modified or changed or deleted. This step is immediately followed by 'Building of Datasets' that I have listed in the 'build/train' step in the scenario.

    I don't understand why Building of the Datasets happen even when the criteria has failed?

    Please let me know if I am missing something here.

    Thanks once again!!!

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,726 Neuron
    Options

    Hi, I use Dataset modified trigger on folders and I do not get executions when the files in the folders have not been changed. So while you assure us that "no file in the folders are modified or changed or deleted" I believe that you are not correct in that statement. Since you have 7 folders it will be hard to see which one is the one triggering the change, so I suggest you create a dummy scenario and test with each of the 7 folder to see which are the ones having changes. Finally please make sure you are not using any of these folders as an output in your flow as you will otherwise be writing to them and hence making a change while your trigger runs which will effectively make your trigger run again.

Setup Info
    Tags
      Help me…