Prepare Recipe infering column data types on creation

PM
PM Registered Posts: 10 ✭✭✭✭

Hi,

when i create a new prepare recipe, it changes the type of my columns

input:

PM_0-1583313231538.png

output:

PM_1-1583313284999.png

Is this the desired behavior? This can cause data loss.

Regards

Answers

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker

    Hi,

    We confirm that this is the desired behavior. DSS has heuristics in place to limit risks of data loss, for example by not using numerical types if there are zero-leading values.

    We're thinking about making this configurable in future releases.

  • PM
    PM Registered Posts: 10 ✭✭✭✭

    But next time when i run the recipe, as the column types were changed from string to int/double, if i receive alphanumeric codes and versions, they will be lost.

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker

    Hi,

    Storage type assignment is only performed once per column, when the column is originally created.

    We never update storage types when running a recipe, and in the specific case of the prepare recipe, we also don't update storage types when designing after the initial column creation.

  • PM
    PM Registered Posts: 10 ✭✭✭✭

    Please, we need this to be configurable in future releases, it is changing all phone fields to numeric, showing warnings at execution time, and losing data.

Setup Info
    Tags
      Help me…