Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
It's probably rather a bug report than a discussion, but there are chances I'm doing something wrong.
I'm calling trunc(MyDateColumn, 'd') on a column and for "2021-10-31T22:00:00.000Z" it returns "2021-10-31T01:00:00.000Z". Obviously, I expect to get "2021-10-31T00:00:00.000Z".
Hi @NikolayK ,
I did a quick test using DSS version 11.0.1 and I'm not able to reproduce this.
Can you share more details? Is this issue happening for all the dates of the column MyDateColumn or only for this specific date? Where are you running the function trunc(MyDateColumn, 'd')?
I did some more extensive testing and the issue seems to be quite subtle. Here's what I did:
1. If I use trunc() in a Prepare recipe, it works fine.
2. It breaks when trunc() is used for a pre-computed column in a Join recipe. What is more, the incumbent dataset is stored in S3 (Parquet), and the other joined dataset is stored in PostgreSQL (not sure if it has any significance). Most of the dates are truncated properly, there are only a few problems.
Let me know if you need more details.
Hi @CatalinaS . After some more work on it, it seems that it's the Join recipe that has a problem. Without any pre-computed columns, it changes "2021-10-31T00:00:00.000Z" to "2021-10-31T01:00:00.000Z".
Thanks for the details provided! I was able to reproduce the issue. Indeed the input date "2021-10-31T00:00:00.000Z" is changed to "2021-10-31T01:00:00.000Z" after a join recipe. The issue doesn't happen if the datetime column is set to be a string rather than a date.
jek.additional.opts = -Duser.timezone="UTC"