Can we monitor any new docker images created by dataiku across the cluster?

Shridhar

As a dataiku admin, I want to keep a track of all the new docker images being created from dataiku. The images are created mainly when we update a code-env with container execution option. So, when we provide access to users. They add few packages to code-env & update to create a new image.

I want to keep a track of any such new images created across the Dataiku cluster. Currently, I'm planning to have a script which will list all docker images and put it in a file. Compare that file with the docker images command every second. If there is any change, it will trigger a notification.

One of the con of above approach is that it would run every sec as cron job consuming resources at machine. Is there any other way to achieve this on Dataiku UI either by scenario/recipe/webapp?

Operating system used: Linux

yonghyun

저도 사용해보지는 않았으나 위와 같은 image를 관리 하는 UI가 있어 보입니다.
해당 메크로를 시나리오로 매시간 반복하면 관리가 될 것으로 보여집니다.

Shridhar

Thank you Yonghyun.

It looks like the picture you've pointed refers to docker images cleanup however I'll explore the option of creating a new macro in a similar fashion to achieve my use-case.

Turribeach

Running something every second will consume a lot of CPU cycles. Why exactly do you need to know when a new image gets created and why do you need to know every second?

ThierryC

Depending on the registry you are using, you may use a webhook to launch your script only when a push happens.

Here is a non-exhaustive list of possible solutions:

• If you are using Docker Hub, the webhook documentation is available here.
• If you are using AWS ECR, you can use EventBridge as described here.
• if you are using Azure Container Registry, you can create a webhook as explained here.

Hope this will help you.

Shridhar

@Turribeach, I'm using an internal repository where image push doesn't happen from dataiku directly. So, we want to trigger a flow the moment any image is created.

I'm looking for an event based trigger for such use-case. I'm currently exploring the option of docker event but I'm still working on optimising this approach because even this needs to run like cron job. Do you have any suggestion here

ThierryC

If you use a webhook, you won't need a cron job, but a small endpoint server that will be called when an image is pushed.
Depending on your infrastructure, the JSON payload will contain metadata to check whether you need to take a specific action for your use case.

Turribeach

Having remembered your previous post it's now more clear what you are trying to do. My advice would be that you really abandon this path that you have taken and look to integrate an image registry repository that you can use freely without restrictions. There is really no need for these images to be uploaded to Artifactory as they are not going to be used by anyone else in the organisation, just Dataiku. And even if you do use Artifactory it can be configured to use basic authentication and allow for seamless image uploads from Dataiku. Failing that use a Cloud registry like Amazon Elastic Container Registry.