We're excited to announce that we're launching the second installment of Dataiku Product Days Register Now

Add hour, minute, and second to shaker increment date processor

0 Kudos

Edit: I found out that the formula processor's inc() function can handle these units, so the value of adding this into the increment date processor as well is primarily just to reduce confusion and improve consistency. I'm fairly certain I'd used the inc() function in the formula processor before, but having forgotten about it and seen the increment date processor right there in the processor library, I assumed it was currently the only way. So this idea is mainly just a UX improvement.



Currently, the increment date processor in shaker recipes can only increment by year, quarter, month, week, and day. But because of leap seconds and daylight savings time, it's common to need to increment by hours, minutes, and seconds as well while preserving calendar accuracy. It would be great if these could be added.

Currently, I have a list of dated events in the future, all in local times. I have another column containing the current time zone for each event, but I need to compute the future time zone for those events due to daylight savings. I have a column containing the end date for a given timezone, and if the date of the event is greater than the end date for the current timezone, I will flip the DST value by subtracting or adding one hour to the converted UTC time based on the nominal timezone. This will allow me to get the true UTC time of any of these events I only have in local times at various locations. The increment date feature would be perfect for this if it supported smaller units. Unfortunately, noninteger values here are rounded down. Python libraries can do the same thing, but these tend to have a lot of overhead for prepare recipes and slow down execution, in addition to adding complexity. Ideally, this could be handled directly by the increment date processor.