Based Image Creation Failed

Erylis
Erylis Registered Posts: 5 ✭✭✭✭

Hello,

I'm trying to build the base image to run on a kubernetes cluster but the command failed. It seems there is a problem with the installation of epel-release.

Could you help me ?

Sending build context to Docker daemon  2.963MB
Step 1/29 : FROM centos:7
 ---> eeb6ee3f44bd
Step 2/29 : WORKDIR /opt/dataiku
 ---> Using cache
 ---> fd72d10177da
Step 3/29 : RUN yum -y install epel-release && yum -y install python2-pip python-devel python3 python3-devel nginx expat zip unzip curl bzip2 freetype libgfortran libgomp procps zeromq-devel libicu-devel libcurl-devel openssl-devel libxml2-devel       && yum -y groupinstall "Development tools" && yum -y autoremove     && yum clean all
 ---> Running in 4e7e09442f37
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: mirror.oxilion.nl
 * extras: mirror.widexs.nl
 * updates: mirror.widexs.nl
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-11 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                Arch             Version         Repository        Size
================================================================================
Installing:
 epel-release           noarch           7-11            extras            15 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 15 k
Installed size: 24 k
Downloading packages:
warning: /var/cache/yum/x86_64/7/extras/packages/epel-release-7-11.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for epel-release-7-11.noarch.rpm is not installed
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-9.2009.0.el7.centos.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : epel-release-7-11.noarch                                     1/1
  Verifying  : epel-release-7-11.noarch                                     1/1

Installed:
  epel-release.noarch 0:7-11

Complete!
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=<repoid> ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>
        or
            subscription-manager repos --disable=<repoid>

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again
The command '/bin/sh -c yum -y install epel-release && yum -y install python2-pip python-devel python3 python3-devel nginx expat zip unzip curl bzip2 freetype libgfortran libgomp procps zeromq-devel libicu-devel libcurl-devel openssl-devel libxml2-devel       && yum -y groupinstall "Development tools" && yum -y autoremove     && yum clean all' returned a non-zero code: 1
Traceback (most recent call last):
  File "/datadrive/dataiku/dataiku-dss-8.0.5/resources/container-exec/build-images.py", line 552, in <module>
    run_wait_check(docker_cmd)
  File "/datadrive/dataiku/dataiku-dss-8.0.5/resources/container-exec/build-images.py", line 458, in run_wait_check
    raise Exception("Command failed: %s - code %s" % (cmd, retcode))
Exception: Command failed: ['docker', 'build', '-t', 'dku-exec-base-vwzr7vesxbk9y3q7tqmm4rmq:dss-8.0.5', '/datadrive/dataiku/dss_data/tmp/exec-docker-base-image.et7ylqfl'] - code 1

Operating system used: ubuntu 18.04

Answers

  • Erylis
    Erylis Registered Posts: 5 ✭✭✭✭

    I found the error, the acces to "mirrors.fedoraproject.org" was denied by the firewall.

  • CoreyS
    CoreyS Dataiker Alumni, Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Registered Posts: 1,150 ✭✭✭✭✭✭✭✭✭

    Thank you for sharing your solution with the rest of the community @Erylis
    !

  • Haithem
    Haithem Partner, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 1 Partner
    edited July 17

    I have the same problem, can you help me to solve the problem :
    
    <DATADIR>
    opmloapp@dvmlodss1-2:/var/opt/data/flat/dataiku/dn$ ./bin/dssadmin build-base-image --type container-exec --with-cuda --tag dataiku-container-exec-base-cuda:11.2 --http-proxy ofr-mlo-dataiku-1-dev:b7f6dbd714@10.156.0.6:8080 
    [+] Saving installation log to /var/opt/data/flat/dataiku/dn/run/install.log
    2022-12-09 19:21:28,881 INFO Building image with options: Namespace(build_from_image=None, copy_to_buildenv=None, cran_mirror='https://cloud.r-project.org', cuda=True, cuda_version='10.0', distrib='centos7', docker_build_opt=None, dockerfile_append=None, dockerfile_prepend=None, http_proxy='ofr-mlo-dataiku-1-dev:b7f6dbd714@10.156.0.6:8080', mode='build', no_proxy=None, py310=False, py37=True, py38=False, py39=False, r=True, source_image=None, source_registry=None, system_packages=None, tag='dataiku-container-exec-base-cuda:11.2', target_registry=None, type='container-exec')
    2022-12-09 19:21:28,882 INFO Preparing build env and Dockerfile
    /opt/application/mloapp/dataiku-dss-11.1.3/resources/container-exec/build-images.py:272: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead
      logging.warn("R not installed in DSS, but trying to install it in container")
    2022-12-09 19:21:28,883 WARNING R not installed in DSS, but trying to install it in container
    2022-12-09 19:21:28,883 INFO Adding R r36 on distrib centos7
    2022-12-09 19:21:28,911 INFO 
    **************************************************************
    * 
    * CUDA Support license
    * 
    * By using this CUDA-enabled images, you agree to the terms
    * of the license agreements for NVIDIA software included in 
    * the images
    *
    * CUDA Toolkit: http://docs.nvidia.com/cuda/eula/index.html
    * cuDNN: https://developer.nvidia.com/cudnn/license_agreement
    *
    **************************************************************
    
    2022-12-09 19:21:29,252 INFO Docker build env and Dockerfile ready, building it
    2022-12-09 19:21:29,252 INFO Build env path:/var/opt/data/flat/dataiku/dn/tmp/exec-docker-base-image.qrbiggua
    2022-12-09 19:21:29,252 INFO Dockerfile content:
    FROM centos:7
    WORKDIR /opt/dataiku
    ENV http_proxy "ofr-mlo-dataiku-1-dev:b7f6dbd714@10.156.0.6:8080"
    
    RUN echo $'[nginx-stable]\n\
    name=nginx stable repo\n\
    baseurl=http://nginx.org/packages/centos/$releasever/$basearch/\n\
    gpgcheck=1\n\
    enabled=1\n\
    gpgkey=https://nginx.org/keys/nginx_signing.key\n\
    module_hotfixes=true' > /etc/yum.repos.d/nginx.repo
    
    RUN yum -y update \
        && yum -y install epel-release \
        && . /etc/os-release && case "$VERSION_ID" in \
            7*) yum -y install procps python-devel python3-devel;; \
            8*) yum -y install procps-ng python2-devel python36-devel ncurses-compat-libs glibc-langpack-en;; \
            *) echo 2>&1 'OS version not supported'; exit 1;; \
           esac \
        && yum -y install curl util-linux bzip2 nginx expat zip unzip freetype libgfortran libgomp libicu-devel libcurl-devel openssl-devel libxml2-devel   \
        && yum -y groupinstall "Development tools" \
        && yum -y autoremove \
        && yum clean all
    
    COPY build-python37.sh build/
    RUN build/build-python37.sh >/tmp/build-python.log && rm -f /tmp/build-python.log
    
    COPY build-r36.sh build/
    RUN . /etc/os-release && case "$VERSION_ID" in \
            7*) yum -y install R-core-devel && yum clean all;; \
            8*) build/build-r36.sh >/tmp/build-r.log && rm -f /tmp/build-r.log;; \
            *) echo 2>&1 'OS version not supported'; exit 1;; \
        esac
    
    RUN echo 'install.packages(c( \
        "httr", \
        "RJSONIO", \
        "dplyr", \
        "curl", \
        "IRkernel", \
        "sparklyr", \
        "ggplot2", \
        "gtools", \
        "tidyr", \
        "rmarkdown", \
        "base64enc", \
        "filelock", \
        "shiny" ), Ncpus=4, repos="https://cloud.r-project.org")' | R --slave --no-restore
    
    # Copy R build capabilities
    COPY install-packages-if-needed.sh build/
    COPY minimal-packages-noconda.txt build/
    COPY irkernel-packages-noconda.txt build/
    
    # Install the equivalent of the base R packages of DSS globally
    RUN mkdir -p R/bin R/R.lib     && rBin=$(command -v R)     && rscriptBin=$(command -v Rscript)     && ln -s "$rBin" "$rscriptBin" R/bin/
    ENV DKU_R_DATAIKU_PACKAGES_PATH=/opt/dataiku/R/R.lib/3.x
    ENV R_LIBS_USER=${DKU_R_DATAIKU_PACKAGES_PATH}
    RUN build/install-packages-if-needed.sh R false build/minimal-packages-noconda.txt https://cloud.r-project.org
    COPY R-exec-wrapper.R R/
    COPY R R/R.lib/
    
    RUN . /etc/os-release && case "$VERSION_ID" in \
            7*) yum install -y yum-utils && \
                yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo && \
                yum clean all;; \
            8*) dnf install -y dnf-plugins-core && \
                dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel8/x86_64/cuda-rhel8.repo && \
                dnf clean all;; \
            *) echo 2>&1 'OS version not supported'; exit 1;; \
        esac
    # For libraries in the cuda-compat-* package: https://docs.nvidia.com/cuda/eula/index.html#attachment-a
    RUN yum install -y \
            cuda-libraries-10-0 \
            cuda-compat-10-0 \
        && ln -s cuda-10.0 /usr/local/cuda \
        && yum clean all
    
    ENV PATH /usr/local/cuda/bin:${PATH}
    
    # EKS requires this in order to expose the CUDA driver in the container
    # https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-gpu.html#gpu-considerations
    ENV NVIDIA_DRIVER_CAPABILITIES=utility,compute
    
    # GKE containers expose the CUDA driver at this location
    # https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#cuda
    ENV PATH=/usr/local/nvidia/bin:$PATH \
        LD_LIBRARY_PATH=/usr/local/nvidia/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}
    ENV CUDNN_VERSION 7.6.5.32
    
    # cuDNN license: https://developer.nvidia.com/cudnn/license_agreement
    RUN CUDNN_DOWNLOAD_SUM=28355e395f0b2b93ac2c83b61360b35ba6cd0377e44e78be197b6b61b4b492ba && \
        curl -OsSL http://developer.download.nvidia.com/compute/redist/cudnn/v7.6.5/cudnn-10.0-linux-x64-v7.6.5.32.tgz && \
        echo "$CUDNN_DOWNLOAD_SUM cudnn-10.0-linux-x64-v7.6.5.32.tgz" | sha256sum -c - && \
        tar --no-same-owner -xzf cudnn-10.0-linux-x64-v7.6.5.32.tgz -C /usr/local --wildcards 'cuda/lib64/libcudnn.so.*' && \
        rm cudnn-10.0-linux-x64-v7.6.5.32.tgz && \
        ldconfig
    
    COPY virtualenv.pyz install-builtin-env-python-packages.sh build/
    COPY dataiku python/dataiku
    COPY dataikuapi python/dataikuapi
    COPY dataikuscoring python/dataikuscoring
    RUN python3.7 build/virtualenv.pyz pyenv && \
        build/install-builtin-env-python-packages.sh pyenv/bin/pip && \
        mkdir -p bin && \
        echo -e '#!/bin/bash -e\nexec /opt/dataiku/pyenv/bin/python "$@' >bin/python && \
        chmod a+x bin/python && \
        bin/python -m compileall -q python && \
        rm -rf ~/.cache/pip
    ENV PYTHONPATH=/opt/dataiku/python
    COPY web/ /opt/dataiku/web/
    COPY resources/nlp /opt/dataiku/resources/nlp/
    
    ENV DKU_SOURCE_LIB_R_PATH=/home/dataiku/lib/instance
    
    WORKDIR /home/dataiku
    
    COPY dss-version.json /opt/dataiku/
    
    # Fake DIP_HOME with tmp folder for R recipes
    ENV DIP_HOME=/home/dataiku/fake_dip_home
    
    RUN groupadd -r dataiku     && useradd -r -g dataiku -d /home/dataiku dataiku     && mkdir fake_dip_home fake_dip_home/tmp lib lib/project lib/instance plugin     && chown -Rh dataiku:dataiku /home/dataiku
    
    # OpenShift compatibility:
    # OpenShift runs containers with an arbitrary uid as an additional security measure
    # Thus, we are not dataiku" and cannot write inside /home/dataiku
    # However, we are always gid=0, so we give /home/dataiku to gid 0 and make sure group can
    # write into it.
    # This is the official recommendation:
    # https://docs.openshift.com/container-platform/4.3/openshift_images/create-images.html#images-create-guide-openshift_create-images
    #  "Support arbitrary user ids"
    #
    # More details:
    # This is enforced through a Security Context Constraint - see
    # https://docs.openshift.com/container-platform/4.3/authentication/managing-security-context-constraints.html
    # One of the SCC says
    #  Run As User Strategy: MustRunAsRange
    #    UID:                    <none>
    #    UID Range Min:              <none>
    #    UID Range Max:              <none>
    # with the range given by an annotation on the project: openshift.io/sa.scc.uid-range=1000540000/10000
    #
    RUN chgrp -R 0 /home/dataiku && chmod -R 775 /home/dataiku
    ENV DKU_CONTAINER_EXEC=1
    
    USER dataiku
    ENTRYPOINT ["/opt/dataiku/bin/python", "-m", "dataiku.container.runner"]
    
    2022-12-09 19:21:29,253 INFO Running command: ['docker', 'build', '-t', 'dataiku-container-exec-base-cuda:11.2', '/var/opt/data/flat/dataiku/dn/tmp/exec-docker-base-image.qrbiggua']
    Sending build context to Docker daemon  21.56MB
    Step 1/43 : FROM centos:7
     ---> eeb6ee3f44bd
    Step 2/43 : WORKDIR /opt/dataiku
     ---> Using cache
     ---> 225dd69eb0da
    Step 3/43 : ENV http_proxy "ofr-mlo-dataiku-1-dev:b7f6dbd714@10.156.0.6:8080"
     ---> Using cache
     ---> 92b73d106853
    Step 4/43 : RUN echo $'[nginx-stable]\nname=nginx stable repo\nbaseurl=http://nginx.org/packages/centos/$releasever/$basearch/\ngpgcheck=1\nenabled=1\ngpgkey=https://nginx.org/keys/nginx_signing.key\nmodule_hotfixes=true' > /etc/yum.repos.d/nginx.repo
     ---> Using cache
     ---> cb3f84ec4ca0
    Step 5/43 : RUN yum -y update     && yum -y install epel-release     && . /etc/os-release && case "$VERSION_ID" in         7*) yum -y install procps python-devel python3-devel;;         8*) yum -y install procps-ng python2-devel python36-devel ncurses-compat-libs glibc-langpack-en;;         *) echo 2>&1 'OS version not supported'; exit 1;;        esac     && yum -y install curl util-linux bzip2 nginx expat zip unzip freetype libgfortran libgomp libicu-devel libcurl-devel openssl-devel libxml2-devel       && yum -y groupinstall "Development tools"     && yum -y autoremove     && yum clean all
     ---> Running in 4ce13d29ad8a
    Loaded plugins: fastestmirror, ovl
    Determining fastest mirrors
    Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
    12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container: (28, 'Connection timed out after 30001 milliseconds')
    
    
     One of the configured repositories failed (Unknown),
     and yum doesn't have enough cached data to continue. At this point the only
     safe thing yum can do is fail. There are a few ways to work "fix" this:
    
         1. Contact the upstream for the repository and get them to fix the problem.
    
         2. Reconfigure the baseurl/etc. for the repository, to point to a working
            upstream. This is most often useful if you are using a newer
            distribution release than is supported by the repository (and the
            packages for the previous distribution release still work).
    
         3. Run the command with the repository temporarily disabled
                yum --disablerepo=<repoid> ...
    
         4. Disable the repository permanently, so yum won't use it by default. Yum
            will then just ignore the repository until you permanently enable it
            again or use --enablerepo for temporary usage:
    
                yum-config-manager --disable <repoid>
            or
                subscription-manager repos --disable=<repoid>
    
         5. Configure the failing repository to be skipped, if it is unavailable.
            Note that yum will try to contact the repo. when it runs most commands,
            so will have to try and fail each time (and thus. yum will be be much
            slower). If it is a very temporary problem though, this is often a nice
            compromise:
    
                yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true
    
    Cannot find a valid baseurl for repo: base/7/x86_64
    The command '/bin/sh -c yum -y update     && yum -y install epel-release     && . /etc/os-release && case "$VERSION_ID" in         7*) yum -y install procps python-devel python3-devel;;         8*) yum -y install procps-ng python2-devel python36-devel ncurses-compat-libs glibc-langpack-en;;         *) echo 2>&1 'OS version not supported'; exit 1;;        esac     && yum -y install curl util-linux bzip2 nginx expat zip unzip freetype libgfortran libgomp libicu-devel libcurl-devel openssl-devel libxml2-devel       && yum -y groupinstall "Development tools"     && yum -y autoremove     && yum clean all' returned a non-zero code: 1
    Traceback (most recent call last):
      File "/opt/application/mloapp/dataiku-dss-11.1.3/resources/container-exec/build-images.py", line 940, in <module>
        run_wait_check(docker_cmd)
      File "/opt/application/mloapp/dataiku-dss-11.1.3/resources/container-exec/build-images.py", line 776, in run_wait_check
        raise Exception("Command failed: %s - code %s" % (cmd, retcode))
    Exception: Command failed: ['docker', 'build', '-t', 'dataiku-container-exec-base-cuda:11.2', '/var/opt/data/flat/dataiku/dn/tmp/exec-docker-base-image.qrbiggua'] - code 1
    
    
    Best Regards,
    HK

Setup Info
    Tags
      Help me…