trigger scenario on file arrives in a subfolder
HeteshPatel
Registered Posts: 11 ✭✭✭✭
hi, wondering if it is possible to trigger a scenario when a file arrives to a a subfolder. I have a network shared folder mounted to a managed folder, the mounted folder is a large network path with multiple subfolders. how do I only point the trigger to a specific subfolder to monitor as opposed the the parent folder?
Tagged:
Best Answer
-
JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 296 Dataiker
Hi @HeteshPatel
,I would suggest pointing a new managed folder to the specific subfolder that you would like to watch and use the "trigger on dataset change" trigger with that managed folder selected.
Thanks!
Jordan
Answers
-
I managed to get this working via a custom code as a trigger, posting here so it may help others.
from dataiku.scenario import Trigger from datetime import datetime import dataiku import glob import os from datetime import timedelta t = Trigger() Folder = dataiku.Folder("<FOLDER ID>") Folder_path = Folder.get_path() #Assuming you have a managed folder and within that folder is your mounted folder input_file = glob.glob(Folder_path + "/" +"MOUNTED_FOLDER/SUBFOLDER" + "/TriggerFile.txt")[0] #Function to check if your file is younger than the specified delta (ideally should match your trigger refresh timing) def is_file_younger_than(file, delta): cutoff = datetime.utcnow() - delta mtime = datetime.utcfromtimestamp(os.path.getmtime(file)) if mtime > cutoff: return True return False if is_file_younger_than(input_file, timedelta(minutes=15)): t.fire()