You now have until September 15th to submit your use case or success story to the 2022 Dataiku Frontrunner Awards!ENTER YOUR SUBMISSION

Managing dependencies across the Pipeline: What are the options?

Level 1
Managing dependencies across the Pipeline: What are the options?


How does one propagate newly introduced dependencies on DEV down the line to TEST and PROD?  Is this a completely manual process?  Are there other mechanisms built into DSS to manage this?  Or, is dependency management across the instances a manual endeavor?

In my organization, we use a typical DEV, TEST, PROD instance infrastructure.  I have used the deployment mechanism in DSS to move projects down the line.  However there are a couple of concerns and questions I have with this approach.

When deploying to TEST or PROD (both automation nodes), I discovered that none of the connections, plugins, or other system dependencies (that may have been added to DEV during the development process) were migrated with the project during the deployment process.  Based on the warnings that pop up, it seems that the lack of dependency movement is intended by the designers of DSS.  Managing this issue (from the DSS Admin perspective) feels a little weighty when you consider the DEV environment may receive new dependencies from any number of developers during the development lifecycle.

In my experience, the issue of dependency propagation would be handled via Jenkins.  Triggered by a change to the repo (Gitlab, Bitbucket, etc.), a Jenkins project would eventually spin up a Docker container to house a temporary environment to handle any manual or automated tests.  The Docker container would originate from an evolving Dockerfile, which would contain all of the system-level dependencies (including a fully-operational instance of DSS).   Any new dependencies would have been added to this Dockerfile prior to the triggering of a Jenkins run (if possible)  

But, even with this approach, the DSS-level dependencies (i.e., new plugins, new connections, etc.) are still needing included if such changes were introduced into DEV.   

Then stepping back from this "typical" experience, I have to wonder how the DSS-model of CI-CD (i.e., using the local deployer) handles this.  At the minimum, why dont at least project dependencies move forward when a DSS project is deployed forward to another automation node?

How are others handling this dependency issue?  Am I missing a glaring piece that would make this issue less weighty?

Operating system used: Linux

0 Kudos
0 Replies