Updating a table via scenario (sharepoint online)
Hello,
I have a dataset made by using the sharepoint online plugin that grabs from a sharepoint list and creates a dataset. The sharepoint gets updated daily and I would like to build a scenario where the dataset is updated when new changes are made to the sharepoint list. I have tried two scenarios: one that updates based on a change to data, and one that simply updates on a timer. Looking at the logs, it appears that it's recognizing some change (I've been testing by adding rows to the sharepoint list) but the dataset in the flow does not update. I've included the scenario settings and steps for each as well as the a snippit of the logs (just one scenario and step - similar for the other one). I've tried messing around with all types of settings and steps. Any ideas if this is possible?
Operating system used: Windows
Answers
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,043 NeuronFirst a word of warning. Scenarios that run on dataset changes can be very expensive in terms of compute, specially if the dataset is a plugin with an API behind it. This means that the only way that Dataiku can know if the dataset has "changed" is by executing the API over and over again and comparing the results in each execution. This sort of functionality is meant to be used on SQL tables where you can execute a quick SQL like a count() or get the latest date and quickly compare the values.
Now looking at your statement I am not sure what exactly is happening. If you define Scenario with a Dataset Change trigger base on the Sharepoint dataset, then you make a change to the dataset and wait, what exactly happens? Does the scenario run or not? Can you post how you set the Dataset Change trigger? I am not sure if this functionality is meant to work with a dataset that is loaded via a plugin which uses an external API.
-
Thank you for the reply. I think we are actually going in a different direction here. Speaking to the dataiku team, it seems as though you are correct, and this stretches beyond its capabilities due to the fact that it's leveraging data from a plugin
-
Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,043 Neuron
Are you aware of editable datasets in Dataiku?
https://doc.dataiku.com/dss/latest/connecting/editable-datasets.html
These should work fine with Scenario triggers. Another option is an Excel file in a network file share which is exposed to Dataiku via a managed folder.