Dataiku K8S Pod unable to connect DSS
[urllib3.connectionpool] http://XXXXX:PORT "POST /dip/api/XXXXX/containers/get-execution HTTP/1.1" 500 None
[ERROR] [root] Could not reach DSS: None: b'Unknown execution context: test_execution_id
Pod config have the necessary variables under the container along with the base image. This execution id is returning as Unknown from the DSS API. Where need to verify it and update to resolve this issue?
Operating system used: Linux
Answers
-
Grixis PartnerApplicant, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 80 ✭✭✭✭✭Hey,
Your id : test_execution_id is valid ?
Do you have more logs or first, could you verify in your kubernetes side if everything is going well ?
It seems your pods isnt configured at all.
Best
-
Hi I got it. The POD is dynamically generating when I execute the tasks from Dataiku. Now the blocker is when I install Conda packages in my code env, lets say test_env, I used force Conda. Its python 3.6. When I install Conda packages, its stuck in the installation process saying "Solving environment: failed with initial frozen solve." When I use build for "None", the whole packages installation works fine. But when I use build for "Container execution" and selected by K8S cluster, I get this error. How to resolve it? Need to add conda-forge in the Docker container anywhere? or any other steps to do? Please help me in detail. I need to install below conda packages in my codeenv,
tensorflow-gpu=2.0.0
tensorboard-data-server
tensorboard-plugin-wit
tensorflow-probability=0.7
tensorflow-estimator=2.0.0
Keras=2.3.1
keras-applications
keras-preprocessingcudatoolkit=10.0.130
cudnn=7.6.5Thanks
-
Grixis PartnerApplicant, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 80 ✭✭✭✭✭
Hey,
Sorry again but I dont get it.
However, to try to help you, first we should identify step by step in which level you have your error. By the way, managing your own custom dockerfile isnt easy and could implies the loose of dataiku support. You should contact them first.
Here is a quick simplified diagram of those interactions you are talking about;
+—+
User
| # initiates
v
Dataiku
| # requests Kubernetes to create a pod
v
K8s Cluster
| # Kubernetes schedules and creates a new pod based on the request from Dataiku
v
Container
| # The pod creates a container using the specified base image and environment variables
| # Within the container, the initialization script runs, including Conda environment setup
v
Code Env
| # Conda environment tries to install specified packages
|
| # ERROR: During Conda package installation, the process fails
| # Error message: "Solving environment: failed"
|
Error: Solving Environment Failed
| # The container returns a 500 error due to the failure in package installation
v
500 Error Returned
| # The 500 error is propagated back from the container through Kubernetes to Dataiku
v
User+—+
So from that,
Verify logs at each level;
- your env config logs in your administration side
- your pod config : kubectl logs <pod_name>
- And in last : Verify the Dockerfile used for the container's image includes your conda-forge config. (In case of you need to add it, ask dataiku support before this operation.)