Hello Dataiku Masters,
I would like to see if there are any scenario runs with cutome trigger or time based trigger i would like to see how many data rows changes after each steps or total no. rows after finishing the entire scenario, possible to see?
Hi @dave. Just to be sure, you want to have access to the history of the dataset changes after each scenario run? For example, would the following screenshot fit your need?
If yes, let me know and I can show you how one member of our team did achieve that using the Dataiku capabilities.
What am really looking is -After each scenario runs how many new records has been overwritten or newly added/appended if its or maybe if we can see by drill down at each job may be under one scenarios so you can have the trend of scenarios to compare the record count? so in nutshell scenario to scenario no. of records added/overwritten etc kind of comparison to understand overall data set behaviour
Hope am able to clear the query.
Hi @dave , yes I think I understand what you are looking for. The screenshot I shared with you is showing one point after each run of the scenario (once a day) for a particular dataset (missing_qa0_tags in the example). That is the visual representation, but the data being plotted can be exported as a dataset, like the file attached which is an excel export of this dataset of metrics.
Each scenario run is identified by the 'timeComputed' column (the time at which the run was made), and for each run you have a set of metrics. For example, 'records:COUNT_RECORDS', which tells you how many records the dataset has after each run, and in this way you could create a trend of home many records were added or removed each day.
I think this information will allow you to get the answer you are looking for. If you think so, I can post a short 'screencast' showing how we are doing it with DSS.