Add hour, minute, and second to shaker increment date processor

natejgardner
natejgardner Neuron, Registered, Neuron 2022, Neuron 2023 Posts: 151 Neuron

Edit: I found out that the formula processor's inc() function can handle these units, so the value of adding this into the increment date processor as well is primarily just to reduce confusion and improve consistency. I'm fairly certain I'd used the inc() function in the formula processor before, but having forgotten about it and seen the increment date processor right there in the processor library, I assumed it was currently the only way. So this idea is mainly just a UX improvement.

--

Currently, the increment date processor in shaker recipes can only increment by year, quarter, month, week, and day. But because of leap seconds and daylight savings time, it's common to need to increment by hours, minutes, and seconds as well while preserving calendar accuracy. It would be great if these could be added.

Currently, I have a list of dated events in the future, all in local times. I have another column containing the current time zone for each event, but I need to compute the future time zone for those events due to daylight savings. I have a column containing the end date for a given timezone, and if the date of the event is greater than the end date for the current timezone, I will flip the DST value by subtracting or adding one hour to the converted UTC time based on the nominal timezone. This will allow me to get the true UTC time of any of these events I only have in local times at various locations. The increment date feature would be perfect for this if it supported smaller units. Unfortunately, noninteger values here are rounded down. Python libraries can do the same thing, but these tend to have a lot of overhead for prepare recipes and slow down execution, in addition to adding complexity. Ideally, this could be handled directly by the increment date processor.

4
4 votes

Released · Last Updated

Hi, In version 12.4, Dataiku updated the 'increment date' processor to allow incrementing by hours, minutes, and seconds in the Prepare recipe. Cheers, Ashley W.

Comments

  • Jason
    Jason Registered Posts: 33 ✭✭✭✭✭

    Yeah +1 on this, I just ran into this same issue. I'm making predictions in minutes... and I need to add the prediction to a known date to get the predicted date. I'd love to have this in a shaker script.

  • Katie
    Katie Dataiker, Registered, Product Ideas Manager Posts: 110 Dataiker

    Thanks for the feedback, this has been added to our backlog and we will keep you updated if anything changes.

    Katie

Setup Info
    Tags
      Help me…