Ready for Dataiku 10? Try out the Crash Course on new features!GET STARTED

DSS Backstage (A New DSS Instance Type)

Background and Introduction:

Dataiku DSS has many different instance types whether it be Dataiku DSS Designer Nodes, Automation Nodes, API Deployer etc. It feels as if there may be a node missing from the Dataiku DSS platform. Specifically one that can be used to help deploy web apps and test out plugins. A place where folks are able to separate their development environment from the designer node which is really considered "production" for plugin developers and web app developers. Since the goal of this instance type would be to cater to the development that happens "behind-the-scenes" for the DSS Designer Node, "Backstage" may be an appropriate name.

Additionally, having a node that caters to plugin development and web app development may also help separate out some of the features of the DSS project page in designer nodes which can sometimes be overwhelming for new DSS users when they try to navigate through the top navigation bar with all of the tools they can use in their project. Perhaps also having a designer node that is more minimal may help on the UX side as well. 

Description (User Story):

As a Dataiku DSS developer I want a new DSS instance type that caters to plugin and web app development so that I have an easier time as a developer versioning, testing and deploying my web apps and plugins to Dataiku DSS designer instances which may act as my "Production environment" but are sort of used today like a development/production all-in-one. 

Example of Enhancements:

I'll know it is successful when the following features/enhancements are implemented for the DSS instance version:

  • The DSS Backstage instance is able to host development containers of which users can SSH into from their local machine. This container may come pre-packaged with all of the dataiku libraries for python, R, etc. so the end-user doesn't need do anything but just start coding. 
  • A CLI tool (perhaps a tool pre-installed in the container described in bullet above) that acts like Yeoman but for building DSS plugins and web apps. 
  • A CI/CD pipeline user interface that enforces best practices for your plugins or in some cases web app by requiring you to test out your plugin programmatically with unit tests that are being ran on a dummy project's flow. 
  • You are able to build web apps in a container (refer to bullet one) and have terminal access to the container so you can easily install JS frontend libraries and use any backend framework (perhaps Django) that you would like. Imagine a Heroku like experience but instead DSS is the one that hosts the web apps. 
  • The ability for an individual developer to provide an SSH key of their own and associate it with themselves via the GUI instead of having to associate it with a security group in DSS which multiple users may belong to. -- perhaps this would be for the designer node.

Priority:

Medium - High 

Note, I realize this will most likely take a while to build. 

Benefit/Impact:

The overall Dataiku DSS experience for both the developers and end users will be greatly enhanced. The reason for this is because if the plugins or web apps themselves are not properly developed by the DSS developer persona (have bugs) then the overall image and experience of Dataiku is not properly reflected since these plugins and web apps help extend the functionality of DSS and what it has to offer. If there is anyway Dataiku can help encourage quality control and best practices then it is a win for everyone.

Like I mentioned before in the background section, this feature may help decouple the DSS project page which  is sometimes overwhelming for DSS users especially new ones. Separating out things like plugin development, web app development may help alleviate this as well as provide an opportunity for these features to grow and be built upon.

Enhancing this experience encourages end users to develop well tested and robust plugins and web apps that they can share with the larger dataiku community. 

Additionally, I will highlight some of the pain points experienced today to give you a sense of why this may be something that is needed. Perhaps shedding light on these pain points may help us construct any alternative solutions if necessary.

What are some of the pain points with creating plugins and web apps in DSS?

It is hard for developers to build more sophisticated web apps in DSS that use JavaScript frontend libraries in Dataiku DSS instances that may be used by multiple users. This is because currently developers need to have server level access to DSS to be able work with JS frameworks which is not always possible to get in a enterprise setting. 

SSH keys for security groups so that developers can push and pull to remote git hosting services isn't something that is easy to configure on the DSS server out of the box. Making it hard to utilize services like GitHub, Bit Bucket, or GitLab through DSS.

When developing plugins and web apps in the designer nodes today it feels as if we are following bad practices by not separating the development environment from our production environment since there is no deployment process built into the development of plugins or web apps. You simply just create it in the designer node from beginning to end.

There is no part in the DSS GUI that enforces or recommends testing of your plugin or web app or perhaps even offer off-the shelf standard tests (scenarios) to help speed up the development processes via standard flow quality control checking.

It is hard to collaborate with other developers on a web app since there is no way to work on different branches from each other (I believe....maybe branches  for the overall project can do this??) slowing down the development process. 

Let me know if there is anything you would like me to clarify or any ideas you would like to add yourself to this. 

Note:

Attached are some of my ideas expressed visually via PDF and PowerPoint. This shows a rough draft skeleton.

 

1 Comment
AshleyW
Dataiker
Dataiker
Status changed to: Acknowledged