Join us on Wednesday, June 3rd for a deep dive into Customer Predictive Analytics Learn more

Prepare Recipe infering column data types on creation

Level 2
Prepare Recipe infering column data types on creation

Hi,

when i create a new prepare recipe, it changes the type of my columns

input:

PM_0-1583313231538.png

output:

PM_1-1583313284999.png

 

Is this the desired behavior? This can cause data loss.

 

Regards

 

0 Kudos
4 Replies
Dataiker
Dataiker

Hi,

We confirm that this is the desired behavior.  DSS has heuristics in place to limit risks of data loss, for example by not using numerical types if there are zero-leading values.

We're thinking about making this configurable in future releases.

0 Kudos
Level 2
Author

But next time when i run the recipe, as the column types were changed from string to int/double, if i receive alphanumeric codes and versions, they will be lost.

0 Kudos
Dataiker
Dataiker

Hi,

Storage type assignment is only performed once per column, when the column is originally created.

We never update storage types when running a recipe, and in the specific case of the prepare recipe, we also don't update storage types when designing after the initial column creation.

0 Kudos
Level 2
Author

Please, we need this to be configurable in future releases, it is changing all phone fields to numeric, showing warnings at execution time, and losing data.