trigger scenario on file arrives in a subfolder

Options
HeteshPatel
HeteshPatel Registered Posts: 11 ✭✭✭✭

hi, wondering if it is possible to trigger a scenario when a file arrives to a a subfolder. I have a network shared folder mounted to a managed folder, the mounted folder is a large network path with multiple subfolders. how do I only point the trigger to a specific subfolder to monitor as opposed the the parent folder?

Tagged:

Best Answer

  • JordanB
    JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 293 Dataiker
    Answer ✓
    Options

    Hi @HeteshPatel
    ,

    I would suggest pointing a new managed folder to the specific subfolder that you would like to watch and use the "trigger on dataset change" trigger with that managed folder selected.

    Thanks!

    Jordan

Answers

  • HeteshPatel
    HeteshPatel Registered Posts: 11 ✭✭✭✭
    edited July 17
    Options

    I managed to get this working via a custom code as a trigger, posting here so it may help others.

    from dataiku.scenario import Trigger
    from datetime import datetime
    import dataiku
    import glob
    import os
    from datetime import timedelta
    
    t = Trigger()
    
    Folder = dataiku.Folder("<FOLDER ID>")
    Folder_path = Folder.get_path()
    
    #Assuming you have a managed folder and within that folder is your mounted folder
    input_file = glob.glob(Folder_path + "/" +"MOUNTED_FOLDER/SUBFOLDER" + "/TriggerFile.txt")[0]
    
    #Function to check if your file is younger than the specified delta (ideally should match your trigger refresh timing)
    def is_file_younger_than(file, delta):
        cutoff = datetime.utcnow() - delta
        mtime = datetime.utcfromtimestamp(os.path.getmtime(file))
        if mtime > cutoff:
            return True
        return False
    
    if is_file_younger_than(input_file, timedelta(minutes=15)):
        t.fire()
Setup Info
    Tags
      Help me…