Prod Deployment Failure - Deployment Template Issue

Harish Kumar

Hi All,

We are using AKS cluster as API node to deploy our model into it.
We are currently using 13.3.3.

Our clusters have gate-keeper policies enabled on them and we need to add securityContext in the deployment template in order to meet the requirements.

Deployment Template:

apiVersion: apps/v1  # Correct API version for Deployment

kind: Deployment

metadata:

  name: __K8S_DEPLOYMENT_ID__  # Specify the actual deployment ID

  # __DKU_LABELS_AND_ANNOTATIONS__

spec:

  replicas: __NB_REPLICAS__  # Specify the number of replicas

  selector:

    matchLabels:

      dataiku.com/dku-apideployer-infra-id: __INFRA_ID__  # Specify the infrastructure ID

      dataiku.com/dku-apideployer-deployment-id: __DEPLOYMENT_ID__  # Specify the deployment ID

  template:

    metadata:

      labels:

        dataiku.com/dku-apideployer-infra-id: __INFRA_ID__

        dataiku.com/dku-apideployer-deployment-id: __DEPLOYMENT_ID__

      # __DKU_LABELS_AND_ANNOTATIONS__

    spec:

      containers:

      - name: apinode-container  # Specify the container name

        image: __IMAGE_TAG__  # Specify the image tag

        ports:

        - containerPort: 12001  # Specify the container port

        securityContext:

          allowPrivilegeEscalation: false  # Set privilege escalation

          readOnlyRootFilesystem: true  # Set the root filesystem as read-only

        env:

          - name: DKU_GRAPHITE_ADDITIONAL_PREFIX

            valueFrom:

              fieldRef:

                fieldPath: metadata.name

          - name: DKU_APIDEPLOYER_K8S_NAME

            valueFrom:

              fieldRef:

                fieldPath: metadata.name

          - name: DKU_APINODE_FAIL_IF_START_ABORTED

            value: "true"

          # Additional environment variables, if necessary

          # __ADDITIONAL_ENV_VARS__

        livenessProbe:

          httpGet:

            path: __SERVICE_ISALIVE_PROBE_PATH__  # Specify the health check endpoint

            port: 12001  # Port for the health check

          initialDelaySeconds: 600  # Delay before the liveness probe starts

          periodSeconds: 50  # Frequency of the liveness probe

        readinessProbe:

          httpGet:

            path: __SERVICE_ISALIVE_PROBE_PATH__  # Specify the health check endpoint

            port: 12001  # Port for the health check

          initialDelaySeconds: 20  # Delay before the readiness probe starts

          periodSeconds: 10  # Frequency of the readiness probe

        volumeMounts:  # Add volume mounts configuration

        - mountPath: /data/dataiku/dss_data  # Path where the volume is mounted

          name: writable-storage             # Corresponding volume name

      volumes:

      - name: writable-storage

        emptyDir: {}

      # __DKU_KUBERNETES_NODESELECTOR__

      # __DKU_KUBERNETES_AFFINITY__

      # __DKU_KUBERNETES_TOLERATIONS__

      # __DKU_KUBERNETES_IMAGE_PULL_SECRETS__

We have added the volumeMounts and volumes to meet the requirements but it fails with below error:

Traceback (most recent call last):  File "/home/dataiku/installdir/scripts/dkuinstall/install_config.py", line 256, in <module>    config.save()  File "/home/dataiku/installdir/scripts/dkuinstall/install_config.py", line 66, in save    with open(self.filename, 'w') as f:OSError: [Errno 30] Read-only file system: '/home/dataiku/data/install.ini'

Any idea on how to fix this issue?

I tried checking the deployment yaml of another app cluster where its able to deploy pods and below is the section for volumeMounts and volumes

        volumeMounts:

        - name: shared-data

          mountPath: /mnt/vaultdata 

        imagePullPolicy: Always

      volumes:

      - name: shared-data 

        emptyDir: {}

      - name: config 

        configMap:

          name: <workload-identity-demo-secrets> # TO UPDATE: update to the name of the configmap in configmap.yml

          items:

          - key: "entry-point.sh"

            path: "entry-point.sh"

Operating system used: Almalinux

Turribeach

Well it's clear to me that this section:

securityContext:   
  readOnlyRootFilesystem: true

it's the one causing the error:

[Errno 30] Read-only file system: '/home/dataiku/data/install.ini'

Dataiku will need to be able to write to /home/dataiku/. It may be possible for you to create a separate volume mount for /home/dataiku/ to avoid this error. But this is a really advanced setup so I will suggest you speak with Dataiku Support or Dataiku Professional Services.

Harish Kumar

https://community.dataiku.com/discussion/comment/45644#Comment_45644

I changed it to /home/dataiku and getting below error

/usr/local/bin/docker-entrypoint.sh: line 7: /home/dataiku/data/bin/python: No such file or directory

Alexandru

Hi Harish,
As mentioned earlier, given the complexity of your setup. Can you open a support ticket along with API deployment diagnostics?
Thanks