Assistance Needed with Custom Python Triggers in Dataiku
Hello Folks,
I recently created a project in Dataiku aimed at collecting metric data at the beginning and end of each month.
Here is a quick summary of my project: I used a scenario to execute an SQL query and set up triggers for the beginning and end of the month, with specific parameters to launch only on working days. However, I am facing an issue where the custom Python triggers are not launching as expected.
Below is the code I used:
Beginning of the Month Code:
from dataiku.scenario import Trigger from datetime import datetime #Date de Today today = datetime.today() #Jour de la semaine weekno = today.weekday() #Heure de today hourday = today.hour #Date du mois daymonth = today.day #Minutes de Today minutes = today.minute t = Trigger() #Run Trigger : Si le 12 est un jour ouvré et pas un week-end et qu'il est 7h30, alors lancer if hourday==16 and minutes==30 and ((weekno <5 and daymonth==1) or (weekno==0 and daymonth==2) or (weekno==0 and daymonth==3)): t.fire() else : pass
End of the Month Code:
from dataiku.scenario import Trigger import calendar from datetime import datetime now = datetime.today() hourday= now.hour daymonth= now.day weekno = now.weekday() _, last_day = calendar.monthrange(now.year, now.month) #Minutes de Today minutes = now.minute if hourday==11 and minutes == 30 and ((weekno<5 and daymonth==last_day) or (weekno==0 and (daymonth==1 or daymonth==2))) : t.fire() else: pass
Despite setting up these conditions, the triggers do not seem to fire correctly. I would greatly appreciate any advice or suggestions to resolve this issue.
Thank you !
Answers
-
Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 321 Neuron
Hi @Shirin
,Check this post about custom triggers for information that may be helpful.
In particular, you can check the backend.log file for trigger logging. This may help you troubleshoot your triggers.
I assume you have tested your logic outside of a trigger to confirm that it works as expected. If not, I would do that next.
Marlan
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,141 Neuron
Marlan suggestion is a good start but I would also add that your code will only fire if the trigger is evaluated in that particular minute of the day. What happens is the server is busy and the trigger is not evaluated in time? There is no garantee it will be evaluated so you need a different solution. Also custom scenario triggers are not ideal since they need to be constantly evaluated. So while one or two might not cause much trouble if you start adding too many of these soon your DSS server will only be constantly evaluating scenario triggers rather than running models.
I would use a different approach. Set 3 timed triggers to run on the 1st, 2nd and 3rd of the month at a time you want, to cover for the potential cases where the 1st or 2nd of the month fall on a weekend. Then define a variable to extract the day of the week and day of the month from the time-based trigger start date time and use a conditional scenario step based execution to check whether the day is not a weekend using a similar expression to the one you had in Python (note Dataiku's day of the week starts on 1 not 0). Note that crucially this uses the expected trigger date time start which will cover cases where your server is under load and can't run your scenario straight away when the trigger start date time is reached. Using now() in the variables may therefore return an incorrect start date time. Also consider that Dataiku scenarios have some catch-up fuctionality so they may run automatically after your server is brought back from a period of downtime.
PS: weekno usually means week number in the year, so I wouldn't use that for that variable name.