Change to editable dataset trigger Scenario?

Options
Marlan
Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 317 Neuron

Hello,

Wondering if anyone has run into changes to an editable dataset not being detected by a Scenario dataset change trigger?

I do find that if I change the dataset settings the scenario will trigger but changing the dataset data doesn't seem to cause the trigger to fire.

If you have run into this, have you figured out any workarounds?

Thanks,

Marlan

Tagged:

Best Answer

  • Keiji
    Keiji Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 52 Dataiker
    Answer ✓
    Options

    Hello @Marlan
    ,

    Thank you so much for your confirmation and response.

    Would you please check the version of your DSS? Support for editable datasets with dataset modification triggers has been introduced since DSS 8.0.3, so you need to use the DSS version 8.0.3 or later to use dataset modification triggers for editable datasets.

    As a side note, dataset modification triggers do not monitor DSS-managed datasets (i.e. datasets that are themselves created by a Flow) by design since dataset modification triggers are designed to watch ​sources in order to trigger the actual build.

    Best regards,
    Keiji

Answers

  • Keiji
    Keiji Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 52 Dataiker
    Options

    Hello,

    For SQL-like datasets such as PostgreSQL, changes in data are not detected by dataset modification triggers. SQL triggers should be used for detecting changes in data in SQL-like datasets.

    For filesystem-like datasets (Uploaded, Filesystem, HDFS), changes in file names, lengths, and last modification time can be detected by dataset modification triggers.

    For the details of dataset modification triggers, please see this reference documentation.

    Best regards,
    Keiji

  • Marlan
    Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 317 Neuron
    Options

    Hi @KeijiY
    ,

    Thanks for the reply.

    I would think that editable datasets would follow the rules of filesystem-like datasets but, based on my testing, they apparently do not. In particular, changes in file length (size) and modification dates do not result in firing of the dataset trigger.

    Adding rows to the editable dataset changes its size in the UI (e.g., the dataset status metric size). Obviously one would logically assume that adding rows would increase the size. However, adding rows does not result in the dataset trigger firing.

    Any sort of data change made via the editing function also results in an update to the Last Modification date shown in the UI. However, changes in the modification date do not result in the dataset trigger firing.

    The dataset trigger WILL fire if changes are made to the editable dataset settings or to the underlying dataset schema (number of columns, column datatypes, etc.).

    Let me know if I'm missing something here. And of course if there is a workaround available to trigger scenarios when editable dataset data changes are made.

    Thanks,

    Marlan

  • Marlan
    Marlan Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Dataiku Frontrunner Awards 2021 Participant, Neuron 2023 Posts: 317 Neuron
    Options

    Thanks @KeijiY
    . We are on the prior version, 8.02. So not a surprise then than editable datasets don't trigger scenarios on that version. Thanks for the clarification.

    Marlan

  • Keiji
    Keiji Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 52 Dataiker
    Options

    Hello @Marlan
    ,

    Thank you so much for your confirmation and response.

    Best regards,
    Keiji

Setup Info
    Tags
      Help me…