How Dataiku reads date columns from a text file.

Options
PANKAJ
PANKAJ Partner, L2 Admin, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered Posts: 26 Partner

When I uploaded two text files in Dataiku in which date columns are there in the format MMddyyyy along with few empty columns. Surprisingly Dataiku reads the date column of the 1st table as Date(Unparsed) type but text type for the second. Now I can easily parse the date column in the 1st table but what about the 2nd one? When I change the type of that from text to date it makes all rows of that column red. So how can I change that into a parsed date format column?

Help, please.

Best Answer

  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Answer ✓
    Options

    Hi @PANKAJ
    ,

    Let's get into some details (if I understand your use case correctly).

    First, you need to understand the difference between "storage type" and "meaning". When in the first table the date column is shown as "Date(Unparsed)" you are talking about the "meaning" that Dataiku is giving to that column. Now, because is (Unparsed) you are still required to do an explicit parsing with a "Prepare Recipe" to actually get Dataiku to understand the column as a "Date", else you won't be able to use the date functions to interact with that column.

    So, even if the second table, for some unknown reason that might be related the contents in one particular cell perhaps, is detected with meaning "Text", you will still need to do the parsing using a "Prepare Recipe" (see this link: https://doc.dataiku.com/dss/latest/preparation/processors/date-parser.html#)

    After you do the parse date step in the prepare recipe, both or your columns will be now given the meaning "Date" by DSS.

    Hope this helps.

Answers

  • CoreyS
    CoreyS Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 1,150 ✭✭✭✭✭✭✭✭✭
    Options

    Hi, @PANKAJ
    ! Can you provide any further details on the thread to assist users in helping you find a solution (insert examples like DSS version etc.) Also, can you let us know if you’ve tried any fixes already?This should lead to a quicker response from the community.

  • PANKAJ
    PANKAJ Partner, L2 Admin, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Registered Posts: 26 Partner
    Options

    Ohh...got it. Thank You so much.

Setup Info
    Tags
      Help me…