Dataset using partition

Options
Ankur5289
Ankur5289 Partner, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts, Registered Posts: 27 Partner
in Using Dataiku

I have a dataset which i need to partition based on timestamp . So there is a column called time stamp and i am using partition to create data set. The partitioning type is Time Range and the range selection is Hour.

This is being created as mongo Db type dataset

i am getting error as below:

An invalid argument has been encountered : Illegal time partitioning value : '2021-05-26T13:37:39.861' with a HOUR partitioning

Has anyone faced similar issue and any leads to solve this - to get the data partitioned based on HOUR time range?

· Share on FacebookShare on Twitter

Answers

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker
    Options

    Hi

    the format of the values expected in the column used as time partitioning source is year-month-day-hour (ie the components of the partitioning value, separated by dashes).

    If the field of the collection you're using doesn't conform to this format, you should create a view to add a derived field with the proper format, and create the DSS dataset on the view, like

    db.createView ("item_view", "item", [{$addFields:{sourceTimestampPartition:{$dateToString:{date:"$sourceTimestamp",format:"%Y-%m-%d-%H"}}}}])

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…