Trigger a scenario based on s3 connection file changes

franbugallo
franbugallo Registered Posts: 2 ✭✭✭

Hi!

I'm trying to set up a download recipe to a managed folder from an s3 connection. The recipe is correctly working but I would like to create an automated trigger so it runs once there has been changes on the files in the s3 connection.

I know that you can create an auto-trigger based on dataset changes. What I'm not able to do is do the auto-trigger based on actual file changes on s3 connection

Is there any way of doing so?

Thanks!!!

Best Answer

  • Zach
    Zach Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 153 Dataiker
    Answer ✓

    Hi @franbugallo
    ,

    Instead of using a download recipe, I recommend configuring your managed folder so that it accesses your S3 bucket directly. This way, the folder will always be up-to-date when you query it, which means that you don't have to use a download recipe. It also supports scenario triggers.


    To configure a managed folder to use S3:

    1. Go to the folder settings, and set the type to Amazon S3.
    2. Set the path to the location of your files within the bucket.8B7E8C98-06FA-4E99-A5C2-4A06F4831045.png


    Now you can add a trigger that will run when the folder contents change. Note that this isn't needed unless you want to run downstream recipes automatically, since the folder itself will always be up-to-date.

    1. Add a 'Trigger on dataset change' trigger. Despite the name, it also works with managed folders.
    2. Add your S3 folder to the trigger. This will only work if the folder is using S3 directly; it won't work with a download recipe.172F68EA-D12E-4125-8A0C-3559FDAE6289_1_201_a.jpeg

    Please let me know if you have any questions.

    Thanks,

    Zach

Answers

Setup Info
    Tags
      Help me…