Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
This is somewhat of an organizational and governance question more than Dataiku, but maybe the "Unified Deployer" in the new 9.0.x version will help me here.
Currently, only DSS admins (me) in our org have the access to migrate projects from Design to our Automation QA and then to Automation Prod instances. While this won't scale well into the future, it has worked fine so far for our needs. Some groups of end users are adamant about controlling their own deployments to QA / Prod. Is there a way to grant individuals the ability to move only their own projects to QA / Prod Automation nodes? I don't want to hold these power-users back, but I'm not sure if I should make them admins of the entire ecosystem.
Does anyone else have a similar situation? How do you handle this?
Some options I'm exploring:
Side note: I refuse to use that deployment macro that has been floating around out there, specifically because of all the "exceptions" for when you can't use it or when it may result in undefined behavior.
Hello,
I would say there are several axis to your topic so maybe not a single way to answer all of them. Here are some thoughts for you:
How do you ensure that a deployment will work, not cause broader issues and ensure that only the right person can do the right thing on the right project?
In DSS, you already have some good features to put in place to make sure projects are tested (think scenarios, metrics & checks or custom code to test your own code ..). As a project is also packaged in a bundle, you are covered on consistency as you are sure to take all you need in one click.
This is a topic on which the new Deployer will definitely help. So you can give it a try. It adds clear security on who can deploy what on what infrastructure but also a way for central teams to see and monitor all projects. The main point of attentions remains on shared objects and configurations (like spark, plugins or connections).
How to you ensure the update is in line with your rules regarding compliance, fairness, explainability?
This is an important topic when dealing with ML projects and potentially critical depending on your company rules and potential regulation. Delegating such tasks is complex and they may represent a fair amount of necessary work in your operationalization process
DSS embeds tools to help you go faster and smarter on that, you can check some hints on Top 3 Dataiku Features for Transparent & Explainable AI. The Model fairness report & Model Document Generator are a great start.
How do you ensure that the right process has been followed for moving to production?
DSS can help you ensure an adequate level consistency in your project, but this broader governance topic needs to be clearly defined internally and communicated to people, that's probably the first task.
You can embed such rules in DSS to some extent using the new Deployer but also using the scenarios (e.g. you can automate deployments of bundles or API services with scenario steps and mix that with other steps, even custom ones to add your own logic).
A few additional thoughts on that:
Hope this helps š
Hi @Taylor,
Good question. Everyone on my team (albeit a fairly small one) does their own deployments. As we add teams to the platform, we are considering adding additional automation instances for each team to use. Currently, we are thinking that we'd continue with each person doing their own deployments.
What is your concern about expanding access to who is able to deploy projects? I'm more just curious. Limiting who can do deployments seems like an eminently reasonable approach and we did discuss doing that a while back but ended up deciding not to do it that way.
Marlan
No, I haven't tried the v9 deployer yet. It sounds like it will give us some additional options on this front which is great.
Coincidentally we are working on a new process to deploy new tables to the production tier of our database platform. We are planning to use scenarios to execute create table scripts and run these under a new service account set up for this purpose. Previous table deployment happened outside of DSS; we are bringing into DSS to simplify the process and align table deployments with project deployments. Currently we are planning for individual to do their own table deployments as part of the project deployments.
Marlan
Hello,
I would say there are several axis to your topic so maybe not a single way to answer all of them. Here are some thoughts for you:
How do you ensure that a deployment will work, not cause broader issues and ensure that only the right person can do the right thing on the right project?
In DSS, you already have some good features to put in place to make sure projects are tested (think scenarios, metrics & checks or custom code to test your own code ..). As a project is also packaged in a bundle, you are covered on consistency as you are sure to take all you need in one click.
This is a topic on which the new Deployer will definitely help. So you can give it a try. It adds clear security on who can deploy what on what infrastructure but also a way for central teams to see and monitor all projects. The main point of attentions remains on shared objects and configurations (like spark, plugins or connections).
How to you ensure the update is in line with your rules regarding compliance, fairness, explainability?
This is an important topic when dealing with ML projects and potentially critical depending on your company rules and potential regulation. Delegating such tasks is complex and they may represent a fair amount of necessary work in your operationalization process
DSS embeds tools to help you go faster and smarter on that, you can check some hints on Top 3 Dataiku Features for Transparent & Explainable AI. The Model fairness report & Model Document Generator are a great start.
How do you ensure that the right process has been followed for moving to production?
DSS can help you ensure an adequate level consistency in your project, but this broader governance topic needs to be clearly defined internally and communicated to people, that's probably the first task.
You can embed such rules in DSS to some extent using the new Deployer but also using the scenarios (e.g. you can automate deployments of bundles or API services with scenario steps and mix that with other steps, even custom ones to add your own logic).
A few additional thoughts on that:
Hope this helps š