Database

[swapnilnavale63]

Hi,

I need to check how frequently my data is updating at database ,Is there any way to check at Dataiku ?

[ben_p · Accepted solution]

Hi @swapnilnavale63 and welcome!

You could use a dataset check to count the number of rows in your database and then log this - what is the expected frequency of changes to the database and are you wanting to trigger some action when it updates?

Ben

[ben_p · Accepted solution]

Hi @swapnilnavale63 and welcome!

You could use a dataset check to count the number of rows in your database and then log this - what is the expected frequency of changes to the database and are you wanting to trigger some action when it updates?

Ben

[Andrey · Accepted solution]

Hi,

Are you interested in some specific table/dataset? On the Dataiku side we track every time your dataset is built, one way to access this information is through the python API, for example:

import dataiku, dataikuapi
ds = dataiku.Dataset('mydataset')
build_history = ds.get_metric_history('Build date')['values']

Another way is to have an additional step that would log information after each time a table was written to.

You can achieve it by creating a table for logging and writing into it from a "Post-write statement" available in the Dataset -> Settings -> Advanced.

 

A third option is similar to the second but on the database side, you can create a database trigger on a table you're interested in and log information each time this table is modified, for example in Postgres:

https://www.postgresql.org/docs/9.1/sql-createtrigger.html

Regards

[swapnilnavale63]

Thank you for your detailed solution !