Submit your inspiring success story or innovative use case to the 2022 Dataiku Frontrunner Awards! ENTER YOUR SUBMISSION

Recommended way to make custom CodeEnv for Python inside managed kubernetes

DrissiReda
Level 3
Recommended way to make custom CodeEnv for Python inside managed kubernetes

When I try making a custom codeEnv for my python notebooks through dataiku. It tries to run the command docker build to do so. I'm running dataiku inside a kubernetes cluster managed by azure (AKS) so docker isn't available. What is the recommended way by dataiku to build such an image. And why doesn't dataiku use something like kaniko to build and push the image to a configured registry.

0 Kudos
5 Replies
HarizoR
Dataiker
Dataiker

Hello DrissiReda,

The recommended way is to avoid running Dataiku itself in a Docker container. It is possible to use it as such, but it comes with a lot of limitations, as detailed in our reference documentation: https://doc.dataiku.com/dss/latest/installation/other/docker.html

In particular, if you run Dataiku in a container, you completely forfeit the ability to leverage containerized execution. 

Hope this helps !

 

Best,

Harizo

0 Kudos
DrissiReda
Level 3
Author

Hello, that is odd. Because on this page about running DSS in AKS: https://doc.dataiku.com/dss/latest/containers/aks/managed.html#create-a-new-containerized-execution-....

The final step is clicking on "Push base images" which uses docker. However AKS doesn't have docker. Maybe I missed something, could you shed some light on this?

0 Kudos
HarizoR
Dataiker
Dataiker

Hi,

This page is about running DSS workloads in AKS, not DSS itself. The goal of containerized execution is to offload heavy computation from recipes or notebooks to Kubernetes pods where they will have access to a bigger pool of resources. In that configuration, the DSS instance still runs on a classical VM, not on Kubernetes.

The base image that you push is not the image of DSS itself but of the "framework" required to execute notebooks and recipes.

 

You can familiarise yourself with the basic concepts by reading this section of the documentation: https://doc.dataiku.com/dss/latest/containers/concepts.html#interaction-between-dss-and-containers

Hope this helps.

Best,

Harizo

DrissiReda
Level 3
Author

Thank you very much. I understand. Any future plans to make DSS itself a bit more container/kubernetes friendly?

0 Kudos
HarizoR
Dataiker
Dataiker

Hi,

As of now the focus is on developing containerized execution, we don't have immediate plans to work on containerizing DSS itself.

Best,

Harizo