Create a code environment manager module

While I don't work for Dataiku and nor I claim to represent them it's clear to me that more advanced / more specific requirements that can easily be implemented using the Dataiku API are not going to make it into the product as they tend to be needed only for certain kind of customers. This same customers tend to have very particular needs even for a specific feature which makes it even more difficult to have one solution that fits all. I believe this is one of the great Dataiku's strengths: a fully featured and rich API to do pretty much anything you can do via the GUI. And reading that you already spent hours doing this work manually I am surprised you haven't tried to automate this already specially knowing from previous posts that you are already familiar with the Dataiku API. So I hope this is the final nudge that you need…

We pretty do all that you mentioned and more using the Dataiku API. In our case we use a combination of Jenkins pipelines, ADO, MS-SQL and PowerBI. Jenkins is our pipeline definition and orchestration tool. We store all Python code environments in Git so we can track changes and deploy them via Jenkins. We use MS-SQL and PowerBI to collect stats on code environment usage, be it both packages and project use.

But the use of these additional tools is merely optional and something that fits within the toolset that we use in our team rather than using them because we need to. You could easily automate all of this and simply use Python code (either on scenarios or recipes), datasets stored in your preferred storage layer and Dataiku Webapps to interact with your code environments.

Dataiku advertises low-code ("clickers") as much as it promotes "coders". In fact that is the key differentiation with other products: Dataiku greatly supports both coders and clickers alike and gives them freedom of choice as it allows the designer to choose what approach to take and not only that but to also mix and match clicking and coding recipes as needed.

We don't see updating code environments as a "housekeeping" task but a fundamental part of our software lifecycle. We provide a Jenkins pipeline for users to deploy Python code environments to Dataiku environments which gives them a no-code self-service solution which gives us full traceability of Who-Is-DOing-What (WIDOW). We also have different pipelines for different use cases and we can permission these pipelines accordingly to provide the right level of access to the right people. None of these users ever need to know or understand the Python APIs.

For a good example of what Dataiku considers housekeeping look at the Macros section in any project.

Furthermore expecting Dataiku to build a predefined SDLC for code environments when most customers will have different needs and requirements will probably end up wth a feature that nearly nobody uses as it will be too specific or too generic. Code environments are key component of your DevOps world as such require handling with integrated and automated approaches which clearly necesites the use of APIs.

Like I said I don't have any authority here but in my experience and knowledge when dealing with Dataiku this is a request that is extremely unlikely to make it to the product for the reasons I stated before. So in my view your only choices are to continue to do things manually or build some tooling to make your life eaiser. Up to you what you think is more productive use of your time.

I never meant to imply that your request was unreasonable or even harder to implement. I also never said that providing an API is the solution to all the user needs. What you do need to realise is that a vendor will have hundreds or even thousands of product enhancement requests many of which will probably be reasonable or quick to implement or even both. As such a prioritisation call needs to happen which will be based on many factors some commercial, some technical, some customer driven like how popular this request is and how many customers are likely to want to use it if it were implemented. In particular I have been told few times by many vendors not just Dataiku that a feature request that can be implemented via a vendor provided API and that it's "niche" (ie that is unlikely to be used by many customers or that it might need a complex design to meet different operational models) will be very unlikely to make it to the product as it can be custom built by the customer using the API to suit a particular need. So the main aim of my post was to make you aware of that point of view and push you to a custom solution since you can clearly benefit from it and gain significant operational efficiency by having such solution.

You can also see that from the vendor point of view providing an API to let customers automate their operational needs is a much more efficient approach than trying to build every possible need into the product. And while I can also agree that many of the automation tooling we have developed ourselves over the years should have been part of the product I can not ignore the fact that any time spent by Dataiku on automation operational needs will be taken away from implementing new features as we know their R&D budget is a finite resource.

Personally I think that letting customers implement their own operational solutions is the correct approach since we often find that any OOTB features around this area don't work exactly the way we want so we may end up doing our own thing anyway. And in the context of what Dataiku is (ie an ML platform) and the expectation of speed and amount of change happening on this space it's clear to me that new features need to be the priority in R&D budget.