How to view how many records changes each steps run in each scenerio?

Options
dave
dave Registered Posts: 17 ✭✭✭✭

Hello Dataiku Masters,

I would like to see if there are any scenario runs with cutome trigger or time based trigger i would like to see how many data rows changes after each steps or total no. rows after finishing the entire scenario, possible to see?

Tagged:

Answers

  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Options

    Hi @dave
    . Just to be sure, you want to have access to the history of the dataset changes after each scenario run? For example, would the following screenshot fit your need?

    Selection_334.png

    If yes, let me know and I can show you how one member of our team did achieve that using the Dataiku capabilities.

    Cheers!

  • dave
    dave Registered Posts: 17 ✭✭✭✭
    Options

    Hello@Igancio,

    What am really looking is -After each scenario runs how many new records has been overwritten or newly added/appended if its or maybe if we can see by drill down at each job may be under one scenarios so you can have the trend of scenarios to compare the record count? so in nutshell scenario to scenario no. of records added/overwritten etc kind of comparison to understand overall data set behaviour

    Hope am able to clear the query.

  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Options

    Hi @dave
    , yes I think I understand what you are looking for. The screenshot I shared with you is showing one point after each run of the scenario (once a day) for a particular dataset (missing_qa0_tags in the example). That is the visual representation, but the data being plotted can be exported as a dataset, like the file attached which is an excel export of this dataset of metrics.

    Each scenario run is identified by the 'timeComputed' column (the time at which the run was made), and for each run you have a set of metrics. For example, 'records:COUNT_RECORDS', which tells you how many records the dataset has after each run, and in this way you could create a trend of home many records were added or removed each day.

    I think this information will allow you to get the answer you are looking for. If you think so, I can post a short 'screencast' showing how we are doing it with DSS.

    Cheers!

  • PK36313
    PK36313 Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 10 Partner
    Options

    Hi @Ignacio_Toledo
    ,

    I am also having similar kind of requirement , I want to tack percentage of increase or decrease in record count every time when the scenario will run .

    I am trying to achieve it though metrics and check but not able to do , because in check we can't get the previous record count.

    Could you please tell me some way out to do this , it would be great help

    Thanks in Advance

Setup Info
    Tags
      Help me…