Database

Solved!
swapnilnavale63
Level 2
Database

Hi,

I need to check how frequently my data is updating at database ,Is there any way to check at Dataiku ?

0 Kudos
2 Solutions
ben_p
Level 5

Hi @swapnilnavale63 and welcome!

You could use a dataset check to count the number of rows in your database and then log this - what is the expected frequency of changes to the database and are you wanting to trigger some action when it updates?

Ben

 

 

View solution in original post

Andrey
Dataiker Alumni

Hi,

Are you interested in some specific table/dataset? On the Dataiku side we track every time your dataset is built, one way to access this information is through the python API, for example:

 

import dataiku, dataikuapi
ds = dataiku.Dataset('mydataset')
build_history = ds.get_metric_history('Build date')['values']

 

 

Another way is to have an additional step that would log information after each time a table was written to.

You can achieve it by creating a table for logging and writing into it from a "" available in the Dataset -> Settings -> Advanced.

Screenshot 2020-07-22 at 11.56.07.png

โ€ƒ

A third option is similar to the second but on the database side, you can create a database trigger on a table you're interested in and log information each time this table is modified, for example in Postgres:

https://www.postgresql.org/docs/9.1/sql-createtrigger.html

 

Regards

 

Andrey Avtomonov
R&D Engineer @ Dataiku

View solution in original post

3 Replies
ben_p
Level 5

Hi @swapnilnavale63 and welcome!

You could use a dataset check to count the number of rows in your database and then log this - what is the expected frequency of changes to the database and are you wanting to trigger some action when it updates?

Ben

 

 

Andrey
Dataiker Alumni

Hi,

Are you interested in some specific table/dataset? On the Dataiku side we track every time your dataset is built, one way to access this information is through the python API, for example:

 

import dataiku, dataikuapi
ds = dataiku.Dataset('mydataset')
build_history = ds.get_metric_history('Build date')['values']

 

 

Another way is to have an additional step that would log information after each time a table was written to.

You can achieve it by creating a table for logging and writing into it from a "" available in the Dataset -> Settings -> Advanced.

Screenshot 2020-07-22 at 11.56.07.png

โ€ƒ

A third option is similar to the second but on the database side, you can create a database trigger on a table you're interested in and log information each time this table is modified, for example in Postgres:

https://www.postgresql.org/docs/9.1/sql-createtrigger.html

 

Regards

 

Andrey Avtomonov
R&D Engineer @ Dataiku
swapnilnavale63
Level 2
Author

Thank you for your detailed solution !

0 Kudos