Sort Recipe is useless

Options
davidmakovoz
davidmakovoz Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2022, Neuron 2023 Posts: 67 Neuron
in Using Dataiku

I'm trying to apply the Sort recipe to a dataset with ~18M rows and 134 columns. The input and output datasets are in a database and the recipe is supposed to run in-database (SQL). I tried different columns to sort by: date, integer, string. I'm always getting this warning message

Warnings were encountered

  • WARNING WARN_RECIPE_SORT_OUTPUT_DS_ORDER_NOT_SUPPORTED
    Output dataset is not compatible with order preservation
    The output dataset type does not support order preservation, it seems that the sort recipe will be useless. More information available in the doc.

I didn't find anything relevant in the documentation.

Please, help.

Tagged:
· Share on FacebookShare on Twitter

Best Answer

  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Answer ✓
    Options

    Hi @davidmakovoz
    , in what kind of dataset are you storing the output? There are some hints at https://doc.dataiku.com/dss/latest/other_recipes/sort.html#write-ordering, and if your output dataset doesn't preserve order writing, you get this message, becase the problem is not the Recipe nor DSS, but the database that is getting the output.

    · Share on FacebookShare on Twitter

Answers

  • Samashti
    Samashti Registered Posts: 1 ✭✭✭
    Options

    Hi Ignacio,

    I am facing the same error but going and fixing the output settings as laid out in the write ordering document is also not helping. could you please detail down the steps to follow?

    · Share on FacebookShare on Twitter
  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Options

    Hi @Samashti

    What kind of connection (or dataset) are you storing the result of the recipe? A filesystem, HDFS, SQL, S3, etc?

    Cheers

    Ignacio

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…