Prod Deployment Failure - Deployment Template Issue

Registered Posts: 7 ✭✭

Hi All,

We are using AKS cluster as API node to deploy our model into it.
We are currently using 13.3.3.

Our clusters have gate-keeper policies enabled on them and we need to add securityContext in the deployment template in order to meet the requirements.

Deployment Template:

apiVersion: apps/v1  # Correct API version for Deployment
kind: Deployment
metadata:
name: __K8S_DEPLOYMENT_ID__ # Specify the actual deployment ID
# __DKU_LABELS_AND_ANNOTATIONS__
spec:
replicas: __NB_REPLICAS__ # Specify the number of replicas
selector:
matchLabels:
dataiku.com/dku-apideployer-infra-id: __INFRA_ID__ # Specify the infrastructure ID
dataiku.com/dku-apideployer-deployment-id: __DEPLOYMENT_ID__ # Specify the deployment ID
template:
metadata:
labels:
dataiku.com/dku-apideployer-infra-id: __INFRA_ID__
dataiku.com/dku-apideployer-deployment-id: __DEPLOYMENT_ID__
# __DKU_LABELS_AND_ANNOTATIONS__
spec:
containers:
- name: apinode-container # Specify the container name
image: __IMAGE_TAG__ # Specify the image tag
ports:
- containerPort: 12001 # Specify the container port
securityContext:
allowPrivilegeEscalation: false # Set privilege escalation
readOnlyRootFilesystem: true # Set the root filesystem as read-only
env:
- name: DKU_GRAPHITE_ADDITIONAL_PREFIX
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: DKU_APIDEPLOYER_K8S_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: DKU_APINODE_FAIL_IF_START_ABORTED
value: "true"
# Additional environment variables, if necessary
# __ADDITIONAL_ENV_VARS__
livenessProbe:
httpGet:
path: __SERVICE_ISALIVE_PROBE_PATH__ # Specify the health check endpoint
port: 12001 # Port for the health check
initialDelaySeconds: 600 # Delay before the liveness probe starts
periodSeconds: 50 # Frequency of the liveness probe
readinessProbe:
httpGet:
path: __SERVICE_ISALIVE_PROBE_PATH__ # Specify the health check endpoint
port: 12001 # Port for the health check
initialDelaySeconds: 20 # Delay before the readiness probe starts
periodSeconds: 10 # Frequency of the readiness probe
volumeMounts: # Add volume mounts configuration
- mountPath: /data/dataiku/dss_data # Path where the volume is mounted
name: writable-storage # Corresponding volume name
volumes:
- name: writable-storage
emptyDir: {}
# __DKU_KUBERNETES_NODESELECTOR__
# __DKU_KUBERNETES_AFFINITY__
# __DKU_KUBERNETES_TOLERATIONS__
# __DKU_KUBERNETES_IMAGE_PULL_SECRETS__

We have added the volumeMounts and volumes to meet the requirements but it fails with below error:

Traceback (most recent call last):  File "/home/dataiku/installdir/scripts/dkuinstall/install_config.py", line 256, in <module>    config.save()  File "/home/dataiku/installdir/scripts/dkuinstall/install_config.py", line 66, in save    with open(self.filename, 'w') as f:OSError: [Errno 30] Read-only file system: '/home/dataiku/data/install.ini'

Any idea on how to fix this issue?

I tried checking the deployment yaml of another app cluster where its able to deploy pods and below is the section for volumeMounts and volumes

        volumeMounts:
- name: shared-data
mountPath: /mnt/vaultdata
imagePullPolicy: Always
volumes:
- name: shared-data
emptyDir: {}
- name: config
configMap:
name: <workload-identity-demo-secrets> # TO UPDATE: update to the name of the configmap in configmap.yml
items:
- key: "entry-point.sh"
path: "entry-point.sh"

Operating system used: Almalinux

Welcome!

It looks like you're new here. Sign in or register to get started.

Answers

  • Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,363 Neuron

    Well it's clear to me that this section:

    securityContext:   
      readOnlyRootFilesystem: true

    it's the one causing the error:

    [Errno 30] Read-only file system: '/home/dataiku/data/install.ini'

    Dataiku will need to be able to write to /home/dataiku/. It may be possible for you to create a separate volume mount for /home/dataiku/ to avoid this error. But this is a really advanced setup so I will suggest you speak with Dataiku Support or Dataiku Professional Services.

  • Registered Posts: 7 ✭✭

    I changed it to /home/dataiku and getting below error

    /usr/local/bin/docker-entrypoint.sh: line 7: /home/dataiku/data/bin/python: No such file or directory

  • Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,269 Dataiker

    Hi Harish,
    As mentioned earlier, given the complexity of your setup. Can you open a support ticket along with API deployment diagnostics?
    Thanks

Welcome!

It looks like you're new here. Sign in or register to get started.

Welcome!

It looks like you're new here. Sign in or register to get started.