Free up storage space on Dataiku

Options
rafael_rosado97
rafael_rosado97 Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 58 Partner

Hi, everyone.

I want to free up storage space on Dataiku disk. I was reading this documentation: https://doc.dataiku.com/dss/latest/operations/disk-usage.html.

So I want to eliminate Job logs through Macro button. At this point I saw another Macro called clear continuous activities logs. What is the difference between these two Macros?

Is it recommended to elimanate these logs in a project in which I'm working currently or only in those that I have finished? I ask this because, what would happen if I want to modify an old proyect in the future?; Will it affect?

Any other option to free up storage space?

Thanks!

Best Answer

  • JordanB
    JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 293 Dataiker
    Answer ✓
    Options

    Hi @rafael_rosado97
    ,

    That is the command we would use to display info about total space and available space on a file system.

    Can you perhaps explain in more specific terms what you are looking for?

    Thanks,

    Jordan

Answers

  • JordanB
    JordanB Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 293 Dataiker
    Options

    Hi @rafael_rosado97
    ,

    Job logs consist of historical data and are useful for understanding job issues. It is safe to remove folders of jobs that are not currently running. Therefore, you can safely run the macro in a project you're currently working on as long as no jobs are running at that time. Once you run the macro, the logs of these jobs will not be available anymore, but the existence of the job will still be registered in the DSS UI.

    Continuous activity logs are specific to continuous activity recipes: https://doc.dataiku.com/dss/latest/streaming/concepts.html#continuous-recipe

    Job logs encompass all of the various other types of jobs and you can specify a max age. For example, below, logs 15 days and older will be cleared.

    Screen Shot 2023-01-26 at 1.45.23 PM.png

    Please let me know if you have any further questions.

    Thanks!

    Jordan

  • rafael_rosado97
    rafael_rosado97 Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 58 Partner
    Options

    It's very clear, @JordanB
    . Thank you so much.

    Just one more question: how can I visualize the available store on the Dataiku disk? I execute the command df -h on the terminal but I would like global information (total space used). Is it possible?

Setup Info
    Tags
      Help me…