Based Image Creation Failed

Erylis
Erylis Registered Posts: 5 ✭✭✭✭
edited July 16 in Setup & Configuration

Hello,

I'm trying to build the base image to run on a kubernetes cluster but the command failed. It seems there is a problem with the installation of epel-release.

Could you help me ?

Sending build context to Docker daemon  2.963MB
Step 1/29 : FROM centos:7
 ---> eeb6ee3f44bd
Step 2/29 : WORKDIR /opt/dataiku
 ---> Using cache
 ---> fd72d10177da
Step 3/29 : RUN yum -y install epel-release && yum -y install python2-pip python-devel python3 python3-devel nginx expat zip unzip curl bzip2 freetype libgfortran libgomp procps zeromq-devel libicu-devel libcurl-devel openssl-devel libxml2-devel       && yum -y groupinstall "Development tools" && yum -y autoremove     && yum clean all
 ---> Running in 4e7e09442f37
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: mirror.oxilion.nl
 * extras: mirror.widexs.nl
 * updates: mirror.widexs.nl
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-11 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                Arch             Version         Repository        Size
================================================================================
Installing:
 epel-release           noarch           7-11            extras            15 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 15 k
Installed size: 24 k
Downloading packages:
warning: /var/cache/yum/x86_64/7/extras/packages/epel-release-7-11.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for epel-release-7-11.noarch.rpm is not installed
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-9.2009.0.el7.centos.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : epel-release-7-11.noarch                                     1/1
  Verifying  : epel-release-7-11.noarch                                     1/1

Installed:
  epel-release.noarch 0:7-11

Complete!
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=<repoid> ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>
        or
            subscription-manager repos --disable=<repoid>

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again
The command '/bin/sh -c yum -y install epel-release && yum -y install python2-pip python-devel python3 python3-devel nginx expat zip unzip curl bzip2 freetype libgfortran libgomp procps zeromq-devel libicu-devel libcurl-devel openssl-devel libxml2-devel       && yum -y groupinstall "Development tools" && yum -y autoremove     && yum clean all' returned a non-zero code: 1
Traceback (most recent call last):
  File "/datadrive/dataiku/dataiku-dss-8.0.5/resources/container-exec/build-images.py", line 552, in <module>
    run_wait_check(docker_cmd)
  File "/datadrive/dataiku/dataiku-dss-8.0.5/resources/container-exec/build-images.py", line 458, in run_wait_check
    raise Exception("Command failed: %s - code %s" % (cmd, retcode))
Exception: Command failed: ['docker', 'build', '-t', 'dku-exec-base-vwzr7vesxbk9y3q7tqmm4rmq:dss-8.0.5', '/datadrive/dataiku/dss_data/tmp/exec-docker-base-image.et7ylqfl'] - code 1

Operating system used: ubuntu 18.04

· Share on FacebookShare on Twitter

Answers

  • Erylis
    Erylis Registered Posts: 5 ✭✭✭✭

    I found the error, the acces to "mirrors.fedoraproject.org" was denied by the firewall.

    · Share on FacebookShare on Twitter
  • CoreyS
    CoreyS Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 1,150 ✭✭✭✭✭✭✭✭✭

    Thank you for sharing your solution with the rest of the community @Erylis
    !

    · Share on FacebookShare on Twitter
  • Haithem
    Haithem Partner, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 1 Partner
    edited July 17 

    I have the same problem, can you help me to solve the problem :

<DATADIR>
opmloapp@dvmlodss1-2:/var/opt/data/flat/dataiku/dn$ ./bin/dssadmin build-base-image --type container-exec --with-cuda --tag dataiku-container-exec-base-cuda:11.2 --http-proxy ofr-mlo-dataiku-1-dev:b7f6dbd714@10.156.0.6:8080 
[+] Saving installation log to /var/opt/data/flat/dataiku/dn/run/install.log
2022-12-09 19:21:28,881 INFO Building image with options: Namespace(build_from_image=None, copy_to_buildenv=None, cran_mirror='https://cloud.r-project.org', cuda=True, cuda_version='10.0', distrib='centos7', docker_build_opt=None, dockerfile_append=None, dockerfile_prepend=None, http_proxy='ofr-mlo-dataiku-1-dev:b7f6dbd714@10.156.0.6:8080', mode='build', no_proxy=None, py310=False, py37=True, py38=False, py39=False, r=True, source_image=None, source_registry=None, system_packages=None, tag='dataiku-container-exec-base-cuda:11.2', target_registry=None, type='container-exec')
2022-12-09 19:21:28,882 INFO Preparing build env and Dockerfile
/opt/application/mloapp/dataiku-dss-11.1.3/resources/container-exec/build-images.py:272: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
  logging.warn("R not installed in DSS, but trying to install it in container")
2022-12-09 19:21:28,883 WARNING R not installed in DSS, but trying to install it in container
2022-12-09 19:21:28,883 INFO Adding R r36 on distrib centos7
2022-12-09 19:21:28,911 INFO 
**************************************************************
* 
* CUDA Support license
* 
* By using this CUDA-enabled images, you agree to the terms
* of the license agreements for NVIDIA software included in 
* the images
*
* CUDA Toolkit: http://docs.nvidia.com/cuda/eula/index.html
* cuDNN: https://developer.nvidia.com/cudnn/license_agreement
*
**************************************************************

2022-12-09 19:21:29,252 INFO Docker build env and Dockerfile ready, building it
2022-12-09 19:21:29,252 INFO Build env path:/var/opt/data/flat/dataiku/dn/tmp/exec-docker-base-image.qrbiggua
2022-12-09 19:21:29,252 INFO Dockerfile content:
FROM centos:7
WORKDIR /opt/dataiku
ENV http_proxy "ofr-mlo-dataiku-1-dev:b7f6dbd714@10.156.0.6:8080"

RUN echo $'[nginx-stable]\n\
name=nginx stable repo\n\
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/\n\
gpgcheck=1\n\
enabled=1\n\
gpgkey=https://nginx.org/keys/nginx_signing.key\n\
module_hotfixes=true' > /etc/yum.repos.d/nginx.repo

RUN yum -y update \
    && yum -y install epel-release \
    && . /etc/os-release && case "$VERSION_ID" in \
        7*) yum -y install procps python-devel python3-devel;; \
        8*) yum -y install procps-ng python2-devel python36-devel ncurses-compat-libs glibc-langpack-en;; \
        *) echo 2>&1 'OS version not supported'; exit 1;; \
       esac \
    && yum -y install curl util-linux bzip2 nginx expat zip unzip freetype libgfortran libgomp libicu-devel libcurl-devel openssl-devel libxml2-devel   \
    && yum -y groupinstall "Development tools" \
    && yum -y autoremove \
    && yum clean all

COPY build-python37.sh build/
RUN build/build-python37.sh >/tmp/build-python.log && rm -f /tmp/build-python.log

COPY build-r36.sh build/
RUN . /etc/os-release && case "$VERSION_ID" in \
        7*) yum -y install R-core-devel && yum clean all;; \
        8*) build/build-r36.sh >/tmp/build-r.log && rm -f /tmp/build-r.log;; \
        *) echo 2>&1 'OS version not supported'; exit 1;; \
    esac

RUN echo 'install.packages(c( \
    "httr", \
    "RJSONIO", \
    "dplyr", \
    "curl", \
    "IRkernel", \
    "sparklyr", \
    "ggplot2", \
    "gtools", \
    "tidyr", \
    "rmarkdown", \
    "base64enc", \
    "filelock", \
    "shiny" ), Ncpus=4, repos="https://cloud.r-project.org")' | R --slave --no-restore

# Copy R build capabilities
COPY install-packages-if-needed.sh build/
COPY minimal-packages-noconda.txt build/
COPY irkernel-packages-noconda.txt build/

# Install the equivalent of the base R packages of DSS globally
RUN mkdir -p R/bin R/R.lib     && rBin=$(command -v R)     && rscriptBin=$(command -v Rscript)     && ln -s "$rBin" "$rscriptBin" R/bin/
ENV DKU_R_DATAIKU_PACKAGES_PATH=/opt/dataiku/R/R.lib/3.x
ENV R_LIBS_USER=${DKU_R_DATAIKU_PACKAGES_PATH}
RUN build/install-packages-if-needed.sh R false build/minimal-packages-noconda.txt https://cloud.r-project.org
COPY R-exec-wrapper.R R/
COPY R R/R.lib/

RUN . /etc/os-release && case "$VERSION_ID" in \
        7*) yum install -y yum-utils && \
            yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo && \
            yum clean all;; \
        8*) dnf install -y dnf-plugins-core && \
            dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo && \
            dnf clean all;; \
        *) echo 2>&1 'OS version not supported'; exit 1;; \
    esac
# For libraries in the cuda-compat-* package: https://docs.nvidia.com/cuda/eula/index.html#attachment-a
RUN yum install -y \
        cuda-libraries-10-0 \
        cuda-compat-10-0 \
    && ln -s cuda-10.0 /usr/local/cuda \
    && yum clean all

ENV PATH /usr/local/cuda/bin:${PATH}

# EKS requires this in order to expose the CUDA driver in the container
# https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-gpu.html#gpu-considerations
ENV NVIDIA_DRIVER_CAPABILITIES=utility,compute

# GKE containers expose the CUDA driver at this location
# https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#cuda
ENV PATH=/usr/local/nvidia/bin:$PATH \
    LD_LIBRARY_PATH=/usr/local/nvidia/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
ENV CUDNN_VERSION 7.6.5.32

# cuDNN license: https://developer.nvidia.com/cudnn/license_agreement
RUN CUDNN_DOWNLOAD_SUM=28355e395f0b2b93ac2c83b61360b35ba6cd0377e44e78be197b6b61b4b492ba && \
    curl -OsSL http://developer.download.nvidia.com/compute/redist/cudnn/v7.6.5/cudnn-10.0-linux-x64-v7.6.5.32.tgz && \
    echo "$CUDNN_DOWNLOAD_SUM cudnn-10.0-linux-x64-v7.6.5.32.tgz" | sha256sum -c - && \
    tar --no-same-owner -xzf cudnn-10.0-linux-x64-v7.6.5.32.tgz -C /usr/local --wildcards 'cuda/lib64/libcudnn.so.*' && \
    rm cudnn-10.0-linux-x64-v7.6.5.32.tgz && \
    ldconfig

COPY virtualenv.pyz install-builtin-env-python-packages.sh build/
COPY dataiku python/dataiku
COPY dataikuapi python/dataikuapi
COPY dataikuscoring python/dataikuscoring
RUN python3.7 build/virtualenv.pyz pyenv && \
    build/install-builtin-env-python-packages.sh pyenv/bin/pip && \
    mkdir -p bin && \
    echo -e '#!/bin/bash -e\nexec /opt/dataiku/pyenv/bin/python "$@' >bin/python && \
    chmod a+x bin/python && \
    bin/python -m compileall -q python && \
    rm -rf ~/.cache/pip
ENV PYTHONPATH=/opt/dataiku/python
COPY web/ /opt/dataiku/web/
COPY resources/nlp /opt/dataiku/resources/nlp/

ENV DKU_SOURCE_LIB_R_PATH=/home/dataiku/lib/instance

WORKDIR /home/dataiku

COPY dss-version.json /opt/dataiku/

# Fake DIP_HOME with tmp folder for R recipes
ENV DIP_HOME=/home/dataiku/fake_dip_home

RUN groupadd -r dataiku     && useradd -r -g dataiku -d /home/dataiku dataiku     && mkdir fake_dip_home fake_dip_home/tmp lib lib/project lib/instance plugin     && chown -Rh dataiku:dataiku /home/dataiku

# OpenShift compatibility:
# OpenShift runs containers with an arbitrary uid as an additional security measure
# Thus, we are not dataiku" and cannot write inside /home/dataiku
# However, we are always gid=0, so we give /home/dataiku to gid 0 and make sure group can
# write into it.
# This is the official recommendation:
# https://docs.openshift.com/container-platform/4.3/openshift_images/create-images.html#images-create-guide-openshift_create-images
#  "Support arbitrary user ids"
#
# More details:
# This is enforced through a Security Context Constraint - see
# https://docs.openshift.com/container-platform/4.3/authentication/managing-security-context-constraints.html
# One of the SCC says
#  Run As User Strategy: MustRunAsRange
#    UID:                    <none>
#    UID Range Min:              <none>
#    UID Range Max:              <none>
# with the range given by an annotation on the project: openshift.io/sa.scc.uid-range=1000540000/10000
#
RUN chgrp -R 0 /home/dataiku && chmod -R 775 /home/dataiku
ENV DKU_CONTAINER_EXEC=1

USER dataiku
ENTRYPOINT ["/opt/dataiku/bin/python", "-m", "dataiku.container.runner"]

2022-12-09 19:21:29,253 INFO Running command: ['docker', 'build', '-t', 'dataiku-container-exec-base-cuda:11.2', '/var/opt/data/flat/dataiku/dn/tmp/exec-docker-base-image.qrbiggua']
Sending build context to Docker daemon  21.56MB
Step 1/43 : FROM centos:7
 ---> eeb6ee3f44bd
Step 2/43 : WORKDIR /opt/dataiku
 ---> Using cache
 ---> 225dd69eb0da
Step 3/43 : ENV http_proxy "ofr-mlo-dataiku-1-dev:b7f6dbd714@10.156.0.6:8080"
 ---> Using cache
 ---> 92b73d106853
Step 4/43 : RUN echo $'[nginx-stable]\nname=nginx stable repo\nbaseurl=http://nginx.org/packages/centos/$releasever/$basearch/\ngpgcheck=1\nenabled=1\ngpgkey=https://nginx.org/keys/nginx_signing.key\nmodule_hotfixes=true' > /etc/yum.repos.d/nginx.repo
 ---> Using cache
 ---> cb3f84ec4ca0
Step 5/43 : RUN yum -y update     && yum -y install epel-release     && . /etc/os-release && case "$VERSION_ID" in         7*) yum -y install procps python-devel python3-devel;;         8*) yum -y install procps-ng python2-devel python36-devel ncurses-compat-libs glibc-langpack-en;;         *) echo 2>&1 'OS version not supported'; exit 1;;        esac     && yum -y install curl util-linux bzip2 nginx expat zip unzip freetype libgfortran libgomp libicu-devel libcurl-devel openssl-devel libxml2-devel       && yum -y groupinstall "Development tools"     && yum -y autoremove     && yum clean all
 ---> Running in 4ce13d29ad8a
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container: (28, 'Connection timed out after 30001 milliseconds')


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=<repoid> ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>
        or
            subscription-manager repos --disable=<repoid>

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot find a valid baseurl for repo: base/7/x86_64
The command '/bin/sh -c yum -y update     && yum -y install epel-release     && . /etc/os-release && case "$VERSION_ID" in         7*) yum -y install procps python-devel python3-devel;;         8*) yum -y install procps-ng python2-devel python36-devel ncurses-compat-libs glibc-langpack-en;;         *) echo 2>&1 'OS version not supported'; exit 1;;        esac     && yum -y install curl util-linux bzip2 nginx expat zip unzip freetype libgfortran libgomp libicu-devel libcurl-devel openssl-devel libxml2-devel       && yum -y groupinstall "Development tools"     && yum -y autoremove     && yum clean all' returned a non-zero code: 1
Traceback (most recent call last):
  File "/opt/application/mloapp/dataiku-dss-11.1.3/resources/container-exec/build-images.py", line 940, in <module>
    run_wait_check(docker_cmd)
  File "/opt/application/mloapp/dataiku-dss-11.1.3/resources/container-exec/build-images.py", line 776, in run_wait_check
    raise Exception("Command failed: %s - code %s" % (cmd, retcode))
Exception: Command failed: ['docker', 'build', '-t', 'dataiku-container-exec-base-cuda:11.2', '/var/opt/data/flat/dataiku/dn/tmp/exec-docker-base-image.qrbiggua'] - code 1


Best Regards,
HK

    · Share on FacebookShare on Twitter
Bookmark this topic

Categories

Setup Info
    Tags
      Help me…