Discover this year's submissions to the Dataiku Frontrunner Awards and give kudos to your favorite use cases and success stories!READ MORE

Trigger a scenario based on s3 connection file changes

Solved!
franbugallo
Level 1
Trigger a scenario based on s3 connection file changes

Hi!

I'm trying to set up a download recipe to a managed folder from an s3 connection. The recipe is correctly working but I would like to create an automated trigger so it runs once there has been changes on the files in the s3 connection.

I know that you can create an auto-trigger based on dataset changes. What I'm not able to do is do the auto-trigger based on actual file changes on s3 connection

Is there any way of doing so?

Thanks!!!

 

 

0 Kudos
1 Solution
ZachM
Dataiker
Dataiker

Hi @franbugallo ,

Instead of using a download recipe, I recommend configuring your managed folder so that it accesses your S3 bucket directly. This way, the folder will always be up-to-date when you query it, which means that you don't have to use a download recipe. It also supports scenario triggers.


To configure a managed folder to use S3:

  1. Go to the folder settings, and set the type to Amazon S3.
  2. Set the path to the location of your files within the bucket.8B7E8C98-06FA-4E99-A5C2-4A06F4831045.png

     


Now you can add a trigger that will run when the folder contents change. Note that this isn't needed unless you want to run downstream recipes automatically, since the folder itself will always be up-to-date.

  1. Add a 'Trigger on dataset change' trigger. Despite the name, it also works with managed folders.
  2. Add your S3 folder to the trigger. This will only work if the folder is using S3 directly; it won't work with a download recipe.172F68EA-D12E-4125-8A0C-3559FDAE6289_1_201_a.jpeg

     

 

Please let me know if you have any questions.

 

Thanks,

Zach

View solution in original post

0 Kudos
2 Replies
ZachM
Dataiker
Dataiker

Hi @franbugallo ,

Instead of using a download recipe, I recommend configuring your managed folder so that it accesses your S3 bucket directly. This way, the folder will always be up-to-date when you query it, which means that you don't have to use a download recipe. It also supports scenario triggers.


To configure a managed folder to use S3:

  1. Go to the folder settings, and set the type to Amazon S3.
  2. Set the path to the location of your files within the bucket.8B7E8C98-06FA-4E99-A5C2-4A06F4831045.png

     


Now you can add a trigger that will run when the folder contents change. Note that this isn't needed unless you want to run downstream recipes automatically, since the folder itself will always be up-to-date.

  1. Add a 'Trigger on dataset change' trigger. Despite the name, it also works with managed folders.
  2. Add your S3 folder to the trigger. This will only work if the folder is using S3 directly; it won't work with a download recipe.172F68EA-D12E-4125-8A0C-3559FDAE6289_1_201_a.jpeg

     

 

Please let me know if you have any questions.

 

Thanks,

Zach

0 Kudos
franbugallo
Level 1
Author

Hi Zach! thank you so much!! This is just was I was looking for.

Thanks again!!