Guidance on Handling _pre_migration_backup_ File

[IsaacChavez]

Hello Dataiku Community,

While performing routine file cleanup on our system, I stumbled upon a backup file named _pre_migration_backup_20220424-210522. This file occupies a significant amount of storage space, and I am contemplating whether it would be safe to delete it. Before proceeding, I wanted to gather insights from the community regarding a couple of questions:

1. How is a file like _pre_migration_backup_20220424-210522 typically generated? I'm curious about the processes or actions within Dataiku DSS that lead to the creation of such pre-migration backup files.

2. Is it safe to delete this backup file, assuming that our system is currently running smoothly and all necessary migrations or updates have been successfully completed since the date of this backup? What precautions should I consider before making a decision on its deletion?

Any guidance or experiences shared regarding the management and implications of removing such backup files would be greatly appreciated. I aim to ensure the optimal use of our storage resources while maintaining system integrity and reliability.

Thank you in advance for your advice and insights.

[tgb417]

@IsaacChavez 

I've deleted some of the really old ones without problems.

However, this is a great topic and I'd love to hear from others what they do with these files.

--Tom

[Turribeach · Accepted solution]

These backup files get generated when you do a DSS server in-place upgrade. You should recognise the date and time of the file which should be when you ran your upgrade. It’s a backup copy of your whole DATA_DIR so it will be as big as that was when it was backed up. You can safely delete this file after the upgrade has been performed and no issues were found. 

If you perform your own backups before upgrading or you have a different roll back strategy you can skip the creation of these backups during upgrades by running DKU_MIGRATE_NOBACKUP=1 prior to the installer upgrade command as shown below. For instance we take an EBS snapshot in our AWS instances so there is no need for us to have this backup file as we can roll back to the snapshot if needed. Your upgrades will run much faster without having to create this backup  

DKU_MIGRATE_NOBACKUP=1 <INSTALL_DIR>/installer.sh ....