Automatic refresh of internal metrics

MRvLuijpen
MRvLuijpen Partner, L2 Admin, L2 Designer, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Frontrunner 2022 Participant, Neuron 2023 Posts: 107 Neuron

Hi,

For a project concerning the usage of DSS, I am making use of the internal metrics (Jobs/Cluster/Commits).

With a scenario, I daily "Build" these metrics, however, it seems that they are not refreshed automatically.

I come to this conclusion because if I do open the internal metrics and refresh the sample, the number of items have increased.

A second thing I noticed is that if I try to use the Metrics on this set, it seems not to be implemented (see screen shot below). Is this correct, or how do I solve this?

Thanks in advance

Best Answer

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker
    Answer ✓

    Hi,

    We confirm that metrics-on-the-metrics-datasets are not implemented.

    However, if you have a recipe that uses one of these internal datasets, and build the target of this recipe (i.e. run this recipe) through a scenario that forces a build, the data will be updated in real time.

Answers

  • MRvLuijpen
    MRvLuijpen Partner, L2 Admin, L2 Designer, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Frontrunner 2022 Participant, Neuron 2023 Posts: 107 Neuron
    Thank you Clément for your response
  • adamnieto
    adamnieto Neuron 2020, Neuron, Registered, Neuron 2021, Neuron 2022, Neuron 2023 Posts: 87 Neuron

    @Clément_Stenac
    , do you know if a "Trigger on dataset change" will work on internal stats datasets when creating a scenario?

    Additionally, are there any API endpoints or classes that can be used get all of the running tasks of DSS? I have used been trying to use the "dataikuapi.dss.future.DSSFuture" class but it doesn't retrieve all of the tasks shown in the /admin/monitoring/background-tasks/ page. This is even when the "all_users=True" parameter is set for "DSSFuture".

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,024 Neuron

    This might be a 3 year too late response but here it goes. The answer to your question is no, a "Trigger on dataset change" on this dataset will not fire when new commits are added. This is because Trigger on dataset change do not look at the data changing just the dataset structure. You can however set up a SQL query trigger over the actual internal table (assuming you are using an external database for your internal stats) and use that for a trigger.

Setup Info
    Tags
      Help me…