The Dataiku Frontrunner Awards are now accepting submissions until July 15 to recognize your achievements! ENTER YOUR SUBMISSION

Delete / Rename files after Sync

Solved!
rmoore
Level 3
Level 3
Delete / Rename files after Sync

I'm setting up a scheduled process to sync .csv files from Azure Blob Storage into Snowflake and am looking for a best practice to rename or delete the .csv files after they've been synchronized with Snowflake so they aren't processed multiple times. 

Is there a recommended way to accomplish this in DSS? 

The files are generated from multiple locations every 10 minutes, so I don't think hourly partitioning is a viable option.

 

Thanks!

0 Kudos
1 Solution
rmoore
Level 3
Level 3
Author

As a follow-up, we were able to accomplish this by creating a custom macro that renames blob storage files as needed. Feel free to message me if anyone would like to discuss details.

View solution in original post

0 Kudos
1 Reply
rmoore
Level 3
Level 3
Author

As a follow-up, we were able to accomplish this by creating a custom macro that renames blob storage files as needed. Feel free to message me if anyone would like to discuss details.

View solution in original post

0 Kudos
Labels (3)
A banner prompting to get Dataiku DSS
Public