Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi Team,
I am using Share drive to read files in Dataiku and i want the files which are older than 30 days to get removed from share drive.
Can we handle this from Dataiku?
Thanks in Advance
Hi,
There is no built-in functionality for this in DSS. Usually, storage providers like S3 would have lifecycle rules you could leverage for this.
If you don't currently have such native functionality you may be able to leverage managed folders API in DSS in a Scenario to delete files older than x days.
import dataiku
import dataikuapi
from datetime import datetime, timedelta
client = dataiku.api_client()
project_key = "DKU_TUTORIAL_FIRSTPLUGIN"
odb_id = "eZBgqcZ3"
folder = dataikuapi.dss.managedfolder.DSSManagedFolder(client, project_key, odb_id)
contents = folder.list_contents()
thirty_days_ago = int((datetime.now() - timedelta(days=30)).timestamp() * 1000)
print(thirty_days_ago)
for item in contents['items']:
if item['lastModified'] < thirty_days_ago:
folder.delete_file(item['path'])