Discover this year's submissions to the Dataiku Frontrunner Awards and give kudos to your favorite use cases and success stories!READ MORE

Cleaning Share Drive

sj0071992
Neuron
Neuron
Cleaning Share Drive

Hi Team,

 

I am using Share drive to read files in Dataiku and i want the files which are older than 30 days to get removed from share drive.

Can we handle this from Dataiku?

 

Thanks in Advance

0 Kudos
1 Reply
AlexT
Dataiker
Dataiker

Hi,

There is no built-in functionality for this in DSS. Usually, storage providers like S3 would have lifecycle rules you could leverage for this. 

If you don't currently have such native functionality you may be able to leverage managed folders API in DSS in a Scenario to delete files older than x days. 

 

import dataiku
import dataikuapi
from datetime import datetime, timedelta

client = dataiku.api_client()
project_key = "DKU_TUTORIAL_FIRSTPLUGIN"
odb_id = "eZBgqcZ3"

folder = dataikuapi.dss.managedfolder.DSSManagedFolder(client, project_key, odb_id)
contents = folder.list_contents()

thirty_days_ago = int((datetime.now() - timedelta(days=30)).timestamp() * 1000)

print(thirty_days_ago)

for item in contents['items']:
    if item['lastModified'] < thirty_days_ago:
        folder.delete_file(item['path'])

  

0 Kudos