Guidance on Handling _pre_migration_backup_ File

IsaacChavez
IsaacChavez Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 14

Hello Dataiku Community,

While performing routine file cleanup on our system, I stumbled upon a backup file named _pre_migration_backup_20220424-210522. This file occupies a significant amount of storage space, and I am contemplating whether it would be safe to delete it. Before proceeding, I wanted to gather insights from the community regarding a couple of questions:

  1. How is a file like _pre_migration_backup_20220424-210522 typically generated? I'm curious about the processes or actions within Dataiku DSS that lead to the creation of such pre-migration backup files.

  2. Is it safe to delete this backup file, assuming that our system is currently running smoothly and all necessary migrations or updates have been successfully completed since the date of this backup? What precautions should I consider before making a decision on its deletion?

Any guidance or experiences shared regarding the management and implications of removing such backup files would be greatly appreciated. I aim to ensure the optimal use of our storage resources while maintaining system integrity and reliability.

Thank you in advance for your advice and insights.

Best Answer

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,024 Neuron
    edited July 17 Answer ✓

    These backup files get generated when you do a DSS server in-place upgrade. You should recognise the date and time of the file which should be when you ran your upgrade. It’s a backup copy of your whole DATA_DIR so it will be as big as that was when it was backed up. You can safely delete this file after the upgrade has been performed and no issues were found.

    If you perform your own backups before upgrading or you have a different roll back strategy you can skip the creation of these backups during upgrades by running DKU_MIGRATE_NOBACKUP=1 prior to the installer upgrade command as shown below. For instance we take an EBS snapshot in our AWS instances so there is no need for us to have this backup file as we can roll back to the snapshot if needed. Your upgrades will run much faster without having to create this backup

    DKU_MIGRATE_NOBACKUP=1 <INSTALL_DIR>/installer.sh ....

Answers

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron

    @IsaacChavez

    I've deleted some of the really old ones without problems.

    However, this is a great topic and I'd love to hear from others what they do with these files.

    --Tom

Setup Info
    Tags
      Help me…