Dataset using partition
I have a dataset which i need to partition based on timestamp . So there is a column called time stamp and i am using partition to create data set. The partitioning type is Time Range and the range selection is Hour.
This is being created as mongo Db type dataset
i am getting error as below:
An invalid argument has been encountered : Illegal time partitioning value : '2021-05-26T13:37:39.861' with a HOUR partitioning
Has anyone faced similar issue and any leads to solve this - to get the data partitioned based on HOUR time range?
Answers
-
Hi
the format of the values expected in the column used as time partitioning source is year-month-day-hour (ie the components of the partitioning value, separated by dashes).
If the field of the collection you're using doesn't conform to this format, you should create a view to add a derived field with the proper format, and create the DSS dataset on the view, like
db.createView ("item_view", "item", [{$addFields:{sourceTimestampPartition:{$dateToString:{date:"$sourceTimestamp",format:"%Y-%m-%d-%H"}}}}])
