Survey banner
The Dataiku Community is moving to a new home! We are temporary in read only mode: LEARN MORE

Problem with Spark enigine

NunziaCoppola
Level 1
Problem with Spark enigine
Hello, 
 
when we read hive tables with engine spark version 2.4.7.7.1.7.2038-1 on CDP version 7.1.7 we have a problem with type date, there's a shift forward of 10 min and 4 s as it is showns in the attached file.
 
Could you help us? 
 
We have already tried to add:
 

spark.sql.legacy.parquet.int96RebaseModeInWrite --> CORRECTED

spark.sql.parquet.datetimeRebaseModeInRead -> CORRECTED

-----------------------------------------------------

spark.sql.legacy.parquet.int96RebaseModeInRead --> CORRECTED

spark.sql.legacy.parquet.int96RebaseModeInWrite --> CORRECTED

-----------------------------------------------------

spark.sql.legacy.parquet.datetimeRebaseModeInWrite --> CORRECTED

spark.sql.legacy.parquet.datetimeRebaseModeInRead --> CORRECTED

spark.sql.legacy.timeParserPolicy --> LEGACY

----------------------------------------------------

without any positive results!

 
Thanks 
 

Operating system used: Red Hat Enterprise 8.6
0 Kudos
0 Replies