Conditions in a output / before writting into a database

chrishnet997
Level 2
Conditions in a output / before writting into a database

So i have a job which basically its a dataset that writes data into a postgresql database. 

The job would be daily so today it would write data with values from todays date, tommorow from tommorows.... 

How do i put a condition in dataiku if there is already data with todays date in the database dont write it again? 

The flow looks like:

Download recipe - folder - table in db 

Thank you 

0 Kudos
1 Reply
AlexT
Dataiker

Hi @chrishnet997 ,

You may want to consider using partitions in this case. 

Before you sync to your SQL create a partitioned file-based ( filesystem or cloud storage) dataset that is partitioned by day. 

https://knowledge.dataiku.com/latest/kb/data-prep/partitions/partitioning-redispatch.html

So the follow could look something like :

Download Recipe - Files in Folder -> Prepare( to truncate date today)   -> Filesystem dataset -> Partitioned dataset and then final sync to database. You can specify the partition to build via a scenario each time to CURRENT_DAY /PREVIOUS_DAY so it only writes data from that particular day to the database. 

Screenshot 2022-05-10 at 13.08.47.png

Let me know if that helps!

0 Kudos

Labels

?
Labels (2)
A banner prompting to get Dataiku