Join us on July 16th as we explore real-world Reinforcement Learning Learn more

write_to_schema or get_dataframe does not keep the same ordering of the dataset

Level 3
write_to_schema or get_dataframe does not keep the same ordering of the dataset

Hi team,

 

we have noticed that reading & writing to SQL tables with the build in functionality of Dataiku does not preserve the same order. This has implications when the order of the dataset plays a role on modelling development processes as cross validation. Thus, it makes it impossible to replicate resutls when the recipe is rerun.

Is this something known to you?

Is there a way to keep the same order of the dataset along the project ?

 

1 Reply
Dataiker
Dataiker

Dear PapaA,

SQL datasets do not preserve the writing order.

The simplest thing to do if order really matters is to use another kind of dataset preserving the order. It can be an S3 bucket, a file, etc.

You can check the dataset type supports ordering by using one instance as the output of a "Sort" recipe. DSS warns you when the dataset type is unable to preserve order.

 

Best regards,

 

Ludovic