Custom script for Container execution to push the base images

Shridhar

I'm running into an error ''' Error while checking kubernetes logs, returned 1 Error from server (BadRequest): container "c" in pod "dataiku-exec-python-afmkmmh" is waiting to start: image can't be pulled '''

The main reason here is because of the access issue for pushing the image to the internal repository. I was checking the custom script option but how do I use it to push the existing image to internal repo.

Any sample 'custom script' for this —> settings - container execution - configuration?

Operating system used: Linux

Operating system used: Linux

Operating system used: Linux

Operating system used: Linux

Operating system used: Linux

Turribeach

Run this command and you will see all the different options. There is an option to push the images using this command:

./bin/dssadmin build-base-image

Shridhar

I've already built the base images and updated that in configuration for container execution under settings. The issue arises when I'm trying to use that option for my python/visual recipe.

The error rises because job creates an image with that tag however it's not able to push to the repo. I wanted to access that image with latest tag so that I can push it to repo.

Above error shows up when I trigger the job.

Turribeach

”I've already built the base images”. How exactly are building the images? Please post the exact command you used. Also rebuilding the images does not need to update any container configuration. Please post a screen shot of your container configuration.

Shridhar

Command run from the data directory

./bin/dssadmin build-base-image --type container-exec
./bin/dssadmin build-base-image --type spark

It's our internal repository, The test to this container configuration is successful as I've pushed images manually to that artifactory location.

Currently, image pre-push hook is none. We can actually use a custom script to push the images to image registry. I'm exploring that option

Turribeach

Can you please add these parameters to your @dssadmin build-base-image --type container-exec@ call:

--mode build-push --target-registry docker.artifacts...

And execute it. You should see the command build the image and then upload it to the repository. You should also see the image tagging have the DSS version your are running.

Shridhar

Thank you for this command Turribeach, I'll try this one however the main problem is that we can't push the images directly to the artifactory. The only way to push the images is through a jenkins job.

So, I thought we could explore the option of using custom script which will somehow trigger this jenkins job.

Turribeach

"the main problem is that we can't push the images directly to the artifactory. The only way to push the images is through a jenkins job." ⇒ This is what you should have started this post as it a crucial point to understand your problem.

Why can't you push them directly using the Dataiku command? Pushing them manually from the local docker cache after you build them successfully should work too but you will need to replicate the image repository name and tagging Dataiku does otherwise the image pull will fail as you can see from your error above. For the image to work you need to push it and tag it as follows:

1. The exec base image must be uploaded to a repository named docker.artifacts.../registry_name/dku-exec-base-[install_id] (without brackets). The install ID can be obtained from DSS_DATA_DIR/install.ini
2. The image must be tagged with the version number matching the DSS version number it was built on (ie 13.4.0). This obviously means every time you upgrade DSS even with a minor upgrade like 13.4.1 you must rebuild, push and tag all the base images again

Another solution is to use a custom named base image but that's more complicated so try with the above first.