Mont/Day Patitioning

pbazin
pbazin Registered Posts: 4 ✭✭✭✭
in Using Dataiku

Hi,

I have a hierarchy of files in a folders organised by month and day (no year).
Therefore my Partitioning is defined as "%M/%D/.*" but seems not to be correct because the list of aprtitions function returns an error.

Detected 0 partitions

Found 57941 unmatched files:

  • /01/01/00939537-c3e2-4f9f-8e09-6d0bfdde4ea8_20:42:18.121400/Report_PredictOnRequest.json
  • /01/01/00b9499a-200c-4c0c-91e5-5823ac4f5131_08:20:30.020730/Report_PredictOnRequest.json
  • /01/01/00be9433-5f4b-4a8b-83d0-54f97b74598e_14:22:20.242891/Report_PredictOnRequest.json
  • ...
  • /12/31/303b8500-00bb-46f4-a0d1-5749286ee407_13:21:34.089887/Report_PredictOnRequest.json

What should I do ?

· Share on FacebookShare on Twitter

Answers

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker

    Hi,

    Dataiku does not support incomplete time hierarchies (month-day without year).

    You will need to manually define two non-time-based partitioning dimensions called "month" and "day", and work manually with them. You will not be able to use time-range management features of DSS.

    · Share on FacebookShare on Twitter
  • pbazin
    pbazin Registered Posts: 4 ✭✭✭✭

    I've got my solution.

    I could have added a "year" folder on top of my "month/day" hierachy but I'm not at liberty to do it.
    What I did is to rename my "month/day" folders as "year-month/day" folders.

    Therefore the partitioning definition becomes "%Y-%M/%D/.*" which works well for DSS.

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…