Cleaning Share Drive

Options
sj0071992
sj0071992 Partner, Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2022, Neuron 2023 Posts: 131 Neuron

Hi Team,

I am using Share drive to read files in Dataiku and i want the files which are older than 30 days to get removed from share drive.

Can we handle this from Dataiku?

Thanks in Advance

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    edited July 17
    Options

    Hi,

    There is no built-in functionality for this in DSS. Usually, storage providers like S3 would have lifecycle rules you could leverage for this.

    If you don't currently have such native functionality you may be able to leverage managed folders API in DSS in a Scenario to delete files older than x days.

    import dataiku
    import dataikuapi
    from datetime import datetime, timedelta
    
    client = dataiku.api_client()
    project_key = "DKU_TUTORIAL_FIRSTPLUGIN"
    odb_id = "eZBgqcZ3"
    
    folder = dataikuapi.dss.managedfolder.DSSManagedFolder(client, project_key, odb_id)
    contents = folder.list_contents()
    
    thirty_days_ago = int((datetime.now() - timedelta(days=30)).timestamp() * 1000)
    
    print(thirty_days_ago)
    
    for item in contents['items']:
        if item['lastModified'] < thirty_days_ago:
            folder.delete_file(item['path'])

Setup Info
    Tags
      Help me…