Join us on July 16th as we explore real-world Reinforcement Learning Learn more

sync project across machines

Level 2
sync project across machines

Dear all, 

I am faced with the task of migrating a number of DSS projects from one instance to another. I have successfully been able to carry out these imports using the dataiku API. However, the problem that I have now is that the target platform is still under construction, so the projects that were migrated are likely to change in the meantime at the source, while adaptations are likely to take place at the target DSS instance. I would like to know if there is a way to use the internal git of the project (or another means) to possibly merge the changes that take place on both instances and that way the developement can continue on both the source and target machines. 

Thanks for your help

0 Kudos
3 Replies
Level 2

Hi Mario,

I found this on their website: https://doc.dataiku.com/dss/latest/collaboration/version-control.html

It looks like you should be able to commit from one DSS instance and pull from another.  I don't have anything to test with, so let us know if that helps.

 

Dataiker
Dataiker

Ciao Mario,

what do you mean exactly with "adaptation" on the target env ? 

If the target env is an automation node you can take advantage of versions: each time you upload a new bundle of the same project you create a new version. You can choose to revert to previous versions if required.

If you are exporting to another DSS design, that is not possible (as there are no versions in design node). 

Generally speaking, the migration flow is one-way and it overwrites the destination (you actually have to delete the project beforehand). Merging between two instances is not supported and furthermore I don't really recommend fiddling with git repos in such way, since you'll most probably break them. 

I suggest to better understand how envs differ: depending on what you are doing on the target env, you might for instance regularly replace the data_dir with a clone coming from the source dss (provided the OS is the same).

I think the best option is to automate the migration. Note that we have a plugin in the store that provides a macro to do exactly this (supports design to automation and design to design), see here. You can deploy it inside a scenario to run it regularly.

Hope that helps, take care.

Omar
Architect @ Dataiku

Level 2
Author

Dear @GreaseMonkey  and @Omar,

Thanks for both your replies. I particularly like the possibility that @GreaseMonkey  pointed to, that is the synchronization via the repo git for the projects, unfortunately the DSS version in the source machine is an early version and it does not seem to provide this option at the project level, and thus I would find myself fiddling with the repos as pointed out by @Omar, so I will avoid that, but if it were a migration between two recent DSS versions, I would certainly try the synchronization via git. In the end I simply imported the project zips and I will keep track of the modifications that happen in the meantime on the source machine in order to implement them at a later time on the target machine, e.g. new recipes, tables, etc.

Regards