Assistance Needed with Custom Python Triggers in Dataiku

[Shirin]

Hello Folks,

I recently created a project in Dataiku aimed at collecting metric data at the beginning and end of each month.

Here is a quick summary of my project: I used a scenario to execute an SQL query and set up triggers for the beginning and end of the month, with specific parameters to launch only on working days. However, I am facing an issue where the custom Python triggers are not launching as expected.

Below is the code I used:

Beginning of the Month Code:

from dataiku.scenario import Trigger
from datetime import datetime

#Date de Today
today = datetime.today()

#Jour de la semaine
weekno = today.weekday()

#Heure de today
hourday = today.hour

#Date du mois
daymonth = today.day

#Minutes de Today
minutes = today.minute

t = Trigger()

#Run Trigger : Si le 12 est un jour ouvré et pas un week-end et qu'il est 7h30, alors lancer

if hourday==16 and minutes==30 and ((weekno <5 and daymonth==1) or (weekno==0 and daymonth==2) or (weekno==0 and daymonth==3)):
    t.fire()
else :
    pass

End of the Month Code:

from dataiku.scenario import Trigger
import calendar
from datetime import datetime

now = datetime.today()
hourday=  now.hour
daymonth=  now.day
weekno = now.weekday()
_, last_day = calendar.monthrange(now.year, now.month)
#Minutes de Today
minutes = now.minute

if hourday==11 and minutes == 30 and ((weekno<5 and daymonth==last_day) or (weekno==0 and (daymonth==1 or daymonth==2))) :
    t.fire()
    
else:
    pass

Despite setting up these conditions, the triggers do not seem to fire correctly. I would greatly appreciate any advice or suggestions to resolve this issue.

Thank you !

[Marlan]

Hi @Shirin,

Check this post about custom triggers for information that may be helpful.

In particular, you can check the backend.log file for trigger logging. This may help you troubleshoot your triggers.

I assume you have tested your logic outside of a trigger to confirm that it works as expected. If not, I would do that next.

Marlan

[Turribeach]

Marlan suggestion is a good start but I would also add that your code will only fire if the trigger is evaluated in that particular minute of the day. What happens is the server is busy and the trigger is not evaluated in time? There is no garantee it will be evaluated so you need a different solution. Also custom scenario triggers are not ideal since they need to be constantly evaluated. So while one or two might not cause much trouble if you start adding too many of these soon your DSS server will only be constantly evaluating scenario triggers rather than running models. 

I would use a different approach. Set 3 timed triggers to run on the 1st, 2nd and 3rd of the month at a time you want, to cover for the potential cases where the 1st or 2nd of the month fall on a weekend. Then define a variable to extract the day of the week and day of the month from the time-based trigger start date time and use a conditional scenario step based execution to check whether the day is not a weekend using a similar expression to the one you had in Python (note Dataiku's day of the week starts on 1 not 0). Note that crucially this uses the expected trigger date time start which will cover cases where your server is under load and can't run your scenario straight away when the trigger start date time is reached. Using now() in the variables may therefore return an incorrect start date time. Also consider that Dataiku scenarios have some catch-up fuctionality so they may run automatically after your server is brought back from a period of downtime.

PS: weekno usually means week number in the year, so I wouldn't use that for that variable name.