Ready for Dataiku 10? Try out the Crash Course on new features!GET STARTED

Cleaning Share Drive

sj0071992
Level 2
Level 2
Cleaning Share Drive

Hi Team,

 

I am using Share drive to read files in Dataiku and i want the files which are older than 30 days to get removed from share drive.

Can we handle this from Dataiku?

 

Thanks in Advance

0 Kudos
1 Reply
AlexT
Dataiker
Dataiker

Hi,

There is no built-in functionality for this in DSS. Usually, storage providers like S3 would have lifecycle rules you could leverage for this. 

If you don't currently have such native functionality you may be able to leverage managed folders API in DSS in a Scenario to delete files older than x days. 

 

import dataiku
import dataikuapi
from datetime import datetime, timedelta

client = dataiku.api_client()
project_key = "DKU_TUTORIAL_FIRSTPLUGIN"
odb_id = "eZBgqcZ3"

folder = dataikuapi.dss.managedfolder.DSSManagedFolder(client, project_key, odb_id)
contents = folder.list_contents()

thirty_days_ago = int((datetime.now() - timedelta(days=30)).timestamp() * 1000)

print(thirty_days_ago)

for item in contents['items']:
    if item['lastModified'] < thirty_days_ago:
        folder.delete_file(item['path'])

  

0 Kudos
A banner prompting to get Dataiku DSS