I need some help to figure out a few things with Dataiku to see if it will work for our needs. We have a different configuration than what Dataiku suggests as far as code environments and containerization, so if "this isn't possible" is the answer, so be it.
Long and short, based on our software development life cycle, we have a series of security scans that we require to be run as part of our pipelines when we're making docker images in general. As a result of this requirement of the SDLC, we have done two things in particular with our installation:
- We've disabled the use of local code environments to prevent unexpected package installations
- We've set up our container making process via Gitlab instead by copying exactly what Dataiku does with its container creation process, and have successfully used these containers on our installation
So the container creation process fits within our SDLC requirements, but we've hit an interesting situation in trying to get Dataiku to understand that there are newer containers available. The example requires some explanation:
Containerized Execution setup:
Config Name: gitlab-test
Image Registry URL: registry.gitlab.com/made-up-company/made-up-docker-folder
Base Name: int:latest
On first Test from the GUI with gitlab-test, Dataiku doesn't find the listed image locally, so it finds registry.gitlab.com/made-up-company/made-up-docker-folder/int:latest (number tag 200001) at the URL, downloads it correctly, and is immediately ready for use in Dataiku.
However, lets say that after that first pull Dataiku does, there's now a NEW version of registry.gitlab.com/made-up-company/made-up-docker-folder/int:latest (number tag 200002) that was made on Gitlab.
When I do the same process as I did for gitlab-test before, Dataiku first looks locally and finds the old image tag 200001 it pulled before using int:latest definition, but doesn't bother to look again at the gitlab registry for new image tag 200002, even though this new tag is now the true int:latest.
Is there a way to change the behavior of Dataiku to instead check the registry first for newer images, or does anyone have any ideas on how to automate docker image pulls via Dataiku?
Thanks!
