Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
I have many datasets from xlsx files and i want to listen on these files if any updated values or new added rows, Or i want to refresh the data sources every period of time to get the updated sheet.
How can i do that?
@mmamdouh First off, welcome to the community. What you are talking about is exactly what scenarios are designed to do. (performing a set of steps when a certain triggering action occurs). I'm presuming that your excel files exist in some location that Dataiku has access to and that you have already created a DSS dataset on top of those files. If that is the case you have some options.
If you are ok to rebuild your dataset on a schedule, you can create a time-based trigger for your scenario (every X min, hours, days) and then add a step to the recipe to to rebuild your dataset. The scenario will trigger on the specified time interval and build your dataset.
If you need to rebuild your dataset only when a change in your excel files is detected, it will be more involved. I'm not of any way to do this out of the box. I think you will probably have to write a custom trigger in Python that looks at those files. This trigger would then check against your files and when it detected a "change" (you'd be responsible for defining that logic) you could trigger the scenario to rebuild your dataset.
@mmamdouh You will need to set up a scenario. To do this, from your flow, select the purple triangle (thats what it looks like in 8.0.0) in the top ribbon bar. This should dropdown a selectable list - which includes "Scenarios". Select that from the menu, then follow these steps (generally).
You can find more specifics in the the documentation here: https://doc.dataiku.com/dss/latest/scenarios/index.html