trigger scenario on file arrives in a subfolder

Solved!
HeteshPatel
Level 2
trigger scenario on file arrives in a subfolder

hi, wondering if it is possible to trigger a scenario when a file arrives to a a subfolder. I have a network shared folder mounted to a managed folder, the mounted folder is a large network path with multiple subfolders. how do I only point the trigger to a specific subfolder to monitor as opposed the the parent folder?

0 Kudos
1 Solution
JordanB
Dataiker

Hi @HeteshPatel,

I would suggest pointing a new managed folder to the specific subfolder that you would like to watch and use the "trigger on dataset change" trigger with that managed folder selected. 

Thanks!

Jordan

View solution in original post

2 Replies
JordanB
Dataiker

Hi @HeteshPatel,

I would suggest pointing a new managed folder to the specific subfolder that you would like to watch and use the "trigger on dataset change" trigger with that managed folder selected. 

Thanks!

Jordan

HeteshPatel
Level 2
Author

I managed to get this working via a custom code as a trigger, posting here so it may help others.

from dataiku.scenario import Trigger
from datetime import datetime
import dataiku
import glob
import os
from datetime import timedelta

t = Trigger()

Folder = dataiku.Folder("<FOLDER ID>")
Folder_path = Folder.get_path()

#Assuming you have a managed folder and within that folder is your mounted folder
input_file = glob.glob(Folder_path + "/" +"MOUNTED_FOLDER/SUBFOLDER" + "/TriggerFile.txt")[0]

#Function to check if your file is younger than the specified delta (ideally should match your trigger refresh timing)
def is_file_younger_than(file, delta):
    cutoff = datetime.utcnow() - delta
    mtime = datetime.utcfromtimestamp(os.path.getmtime(file))
    if mtime > cutoff:
        return True
    return False

if is_file_younger_than(input_file, timedelta(minutes=15)):
    t.fire()