Custom python trigger time-conditioned won't launch Scenario

Harry
Harry Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 9
edited July 16 in Using Dataiku

Hello,
I am confronted in the same problem.

I have create a custpm trigger:

from dataiku.scenario import Trigger
from datetime import datetime, date
import dataiku

from loguru import logger

# Run every seconds: 10800 = 3600 x 3 = 3 hours

t = Trigger()
execution_date = date.today()
logger.info(f"execution_date: {execution_date}")
now = datetime.today()
logger.info(f"now: {now}")
execution_hour = now.hour
logger.info(f"execution_hour: {execution_hour}")

date_format = "%d-%m-%Y"

update_date_str = dataiku.get_custom_variables().get("update_date")
if update_date_str:
    update_date = datetime.strptime(update_date_str, date_format)
    logger.info(f"update_date: {update_date}")
else:
    update_date = (datetime.fromtimestamp(0)).date()
    logger.info(f"update_date: {update_date}")
    
delta_days = execution_date - update_date
logger.info(f"delta_days: {delta_days}")

# lire la date d'update stockée en tant que variable globale
weekday_execution = execution_date.weekday()
logger.info(f"weekday_execution: {weekday_execution}")
if weekday_execution < 2: # lundi 0, mardi 1   value: 2
    # 5 execution par jour 8,11,14,17,20
    if execution_hour >= 8 and execution_hour =< 20:  
        if delta_days > 3: 
            t.fire()

i have added a step : Build only this dataset

first test: Run every seconds : 60, la the scenario was never launched (Auto-triggers: on and custom trigger: on)

second test: I have added the second Time based trigger and activate it. The script launched even the condition in the custom trigger not verified.

can someone help me ?

Thanks

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,112 Neuron

    Thanks for posting again, much easier to reproduce now. OK the problem is that your code has some errors:

    Screenshot 2024-04-26 at 10.14.09.png

    It should be "<=" not "=<". It's highly advisable that you test your custom Python trigger code in a Jupyter Notebook or at least you review the Scenario logs to catch any Python errors. With regards to the the second Time based trigger you added you should note all scenario triggers are evaluated individually so currently there is no option to combine them into a logical criteria. I have actually raised an enhancement request for this so feel free to vote for it by clicking on the up arrow:

    https://community.dataiku.com/t5/Product-Ideas/Add-support-for-logically-combining-triggers-in-Scenarios/idi-p/40656

  • Harry
    Harry Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 9
    edited July 17

    Thank you for the answer

    from dataiku.scenario import Trigger
    from datetime import datetime, date
    import dataiku
    
    from loguru import logger
    
    now = datetime.today()
    execution_date = now.replace(hour=0, minute=0, second=0, microsecond=0)
    logger.info(f"execution_date: {execution_date}")
    execution_hour = now.hour
    logger.info(f"execution_hour: {execution_hour}")
    update_date = now.replace(day=1, hour=0, minute=0, second=0, microsecond=0)
    logger.info(f"update_date: {update_date}")
    
    delta_days = (execution_date - update_date).days
    logger.info(f"delta_days: {delta_days}")
    
    # lire la date d'update stockée en tant que variable globale
    weekday_execution = execution_date.weekday()
    logger.info(f"weekday_execution: {weekday_execution}")
    if weekday_execution < 5: # lundi 0, mardi 1   value: 2
        # 5 execution par jour 8,11,14,17,20
        if execution_hour >= 8 and execution_hour <= 20:  
            if delta_days > 3: 
                print("Fire")

    I have fix the code. Now i try to run the scenarion to populate the table using the custom trigger every 300 seconds. The scenario never launched. How can I launch it automatically ?
    Capture.PNG

    Thanks

    Regars

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,112 Neuron

    I removed the t.fire() as I was testing on Notebook. You should put it back in, that's the statement that fires the trigger.

  • Harry
    Harry Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 9
    edited July 17

    Yes, i have replaced the print with t.fire(). but the trigger not launch automatically

    The code used is below:

    from dataiku.scenario import Trigger
    from datetime import datetime, date
    import dataiku
    
    from loguru import logger
    
    # Run every seconds: 10800 = 3600 x 3 = 3 hours
    
    t = Trigger()
    now = datetime.today()
    logger.info(f"now: {now}")
    execution_hour = now.hour
    logger.info(f"execution_hour: {execution_hour}")
    
    execution_date = now.replace(hour=0, minute=0, second=0, microsecond=0)
    logger.info(f"execution_date: {execution_date}")
    
    date_format = "%d-%m-%Y"
    
    update_date_str = dataiku.get_custom_variables().get("update_date")
    if update_date_str:
        update_date = datetime.strptime(update_date_str, date_format)
        logger.info(f"update_date: {update_date}")
    else:
        update_date = (datetime.fromtimestamp(0)).date()
        logger.info(f"update_date: {update_date}")
    
    delta_days = (execution_date - update_date).days
    logger.info(f"delta_days: {delta_days}")
    
    # lire la date d'update stockée en tant que variable globale
    weekday_execution = execution_date.weekday()
    logger.info(f"weekday_execution: {weekday_execution}")
    if weekday_execution < 2: # lundi 0, mardi 1   value: 2
        # 5 execution par jour 8,11,14,17,20
        if execution_hour >= 8 and execution_hour <= 20:  
            if delta_days > 3: 
                t.fire()

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,112 Neuron

    The code is only meant to execute the trigger on Monday and Tuesday between 8 and 20hs. Not sure why you expect it to run today which is a Friday. Also note the datetime object will be based on the Dataiku's server date time, which may be different than your local date time.

  • Harry
    Harry Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 9
    edited July 17

    Thanks
    i have tested by modifiying this conditions:

    if weekday_execution < 6: # lundi 0, mardi 1   value: 2
        # 5 execution par jour 8,11,14,17,20
        if execution_hour >= 8 and execution_hour <= 20:  
            if delta_days > 3: 
                t.fire()

    That means that the trigger is executed all the week except sunday between 8 and 20 o'clock.
    The trigger is not launched

    Thanks.

    Regards

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,112 Neuron

    Not really at it depends on the delta_days which depends on a project variable. You are logging of all these values with your logger.info() calls so I am surprised you haven't looked at the execution logs to see the values of all different variables of your code. My guess is that your update_date_str is empty hence update_date defaults to 1970-01-01 and delta_days is huge.

  • Harry
    Harry Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 9
    edited July 17

    The output of my logger when i'am testing using notebook.

    Unfortunatly, i'm not admin to see the log of the trigger.

    2024-04-26 15:29:00.467 | INFO     | __main__:<module>:9 - execution_date: 2024-04-26 00:00:00
    2024-04-26 15:29:00.469 | INFO     | __main__:<module>:11 - execution_hour: 15
    2024-04-26 15:29:00.470 | INFO     | __main__:<module>:13 - update_date: 2024-04-01 00:00:00
    2024-04-26 15:29:00.472 | INFO     | __main__:<module>:16 - delta_days: 25
    2024-04-26 15:29:00.473 | INFO     | __main__:<module>:20 - weekday_execution: 4

    Regards

Setup Info
    Tags
      Help me…