[2020/12/17-16:34:42.738] [ActivityExecutor-32] [INFO] [dku] running compute_Python27_NP - ---------------------------------------- [2020/12/17-16:34:42.738] [ActivityExecutor-32] [INFO] [dku] running compute_Python27_NP - DSS startup: jek version:8.0.2 [2020/12/17-16:34:42.738] [ActivityExecutor-32] [INFO] [dku] running compute_Python27_NP - DSS home: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data [2020/12/17-16:34:42.738] [ActivityExecutor-32] [INFO] [dku] running compute_Python27_NP - OS: Linux 3.10.0-1062.18.1.el7.x86_64 amd64 - Java: Oracle Corporation 1.8.0_242 [2020/12/17-16:34:42.737] [ActivityExecutor-32] [INFO] [dku.flow.jobrunner] running compute_Python27_NP - Allocated a slot for this activity! [2020/12/17-16:34:42.738] [ActivityExecutor-32] [INFO] [dku.flow.jobrunner] running compute_Python27_NP - Run activity [2020/12/17-16:34:42.754] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Executing default pre-activity lifecycle hook [2020/12/17-16:34:42.836] [ActivityExecutor-32] [DEBUG] [dku.datasets.hdfs] running compute_Python27_NP - HDFS dataset handler dataset=SERVERCONFIGURATION.Python27 connection=hdfs_sentry cpr=/sentry/bhbwg/design resolvedPath=/SERVERCONFIGURATION/Python27 connRootSA=nullconnRootWithinSA=/sentry/bhbwg/design configuredRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 effectiveRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:42.841] [ActivityExecutor-32] [DEBUG] [dku.fsproviders.hdfs] running compute_Python27_NP - Build HDFSProvider conn=hdfs_sentry cpr=/sentry/bhbwg/design pWCR=/SERVERCONFIGURATION/Python27 crSA=null crWSA=/sentry/bhbwg/design rpWSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:42.843] [ActivityExecutor-32] [DEBUG] [dku.hadoop] running compute_Python27_NP - Initializing Hadoop FS with context UGI: cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS) (login: cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)) rootPathURI=null [2020/12/17-16:34:42.869] [ActivityExecutor-32] [INFO] [dku.hadoop] running compute_Python27_NP - Forcing disable of defaultScheme cache (#8388): hdfs [2020/12/17-16:34:42.888] [ActivityExecutor-32] [DEBUG] [org.apache.htrace.core.Tracer] running compute_Python27_NP - sampler.classes = ; loaded no samplers [2020/12/17-16:34:42.891] [ActivityExecutor-32] [DEBUG] [org.apache.htrace.core.Tracer] running compute_Python27_NP - span.receiver.classes = ; loaded no span receivers [2020/12/17-16:34:43.666] [ActivityExecutor-32] [WARN] [org.apache.hadoop.util.NativeCodeLoader] running compute_Python27_NP - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [2020/12/17-16:34:43.672] [ActivityExecutor-32] [WARN] [org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory] running compute_Python27_NP - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. [2020/12/17-16:34:43.683] [ActivityExecutor-32] [DEBUG] [dku.fsproviders.hdfs] running compute_Python27_NP - Built Hadoop FS for: null -> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-439468239_32, ugi=cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)]] [2020/12/17-16:34:43.904] [ActivityExecutor-32] [INFO] [dku.datasets.hdfs.permissions] running compute_Python27_NP - Setting gateway ACL (create): SERVERCONFIGURATION.Python27 [2020/12/17-16:34:43.905] [ActivityExecutor-32] [INFO] [dku.fsproviders.hdfs] running compute_Python27_NP - ACL synchronization disabled, noop [2020/12/17-16:34:43.905] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Checking if sources are ready [2020/12/17-16:34:43.914] [ActivityExecutor-32] [DEBUG] [dku.db.internal] running compute_Python27_NP - Created DSSDBConnection dssdb-h2-flow_state-CTEDeuX [2020/12/17-16:34:43.924] [ActivityExecutor-32] [DEBUG] [dku.dataset.hash] running compute_Python27_NP - Readiness cache miss for dataset__cabhbwg__SERVERCONFIGURATION.f_loan_data__NP [2020/12/17-16:34:43.932] [ActivityExecutor-32] [INFO] [dku.datasets.file] running compute_Python27_NP - Building Filesystem handler config: {"path":"/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/uploads/SERVERCONFIGURATION/datasets/f_loan_data","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]}} [2020/12/17-16:34:43.934] [ActivityExecutor-32] [INFO] [dku.datasets.ftplike] running compute_Python27_NP - Enumerating Filesystem dataset prefix= [2020/12/17-16:34:43.936] [ActivityExecutor-32] [DEBUG] [dku.fs.local] running compute_Python27_NP - Enumerating local filesystem prefix=/ [2020/12/17-16:34:43.938] [ActivityExecutor-32] [DEBUG] [dku.fs.local] running compute_Python27_NP - Enumeration done nb_paths=1 size=1113566 [2020/12/17-16:34:43.939] [ActivityExecutor-32] [INFO] [dku.dataset.hash] running compute_Python27_NP - Caching readiness for dataset__cabhbwg__SERVERCONFIGURATION.f_loan_data__NP s=READY h=0kXinyc4XcT0QCON65UC9Q [2020/12/17-16:34:43.940] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Checked source readiness SERVERCONFIGURATION.f_loan_data -> true [2020/12/17-16:34:43.940] [ActivityExecutor-32] [DEBUG] [dku.flow.activity] running compute_Python27_NP - Computing hashes to propagate BEFORE activity [2020/12/17-16:34:43.943] [ActivityExecutor-32] [DEBUG] [dku.dataset.hash] running compute_Python27_NP - Readiness cache miss for dataset__cabhbwg__SERVERCONFIGURATION.f_loan_data__NP [2020/12/17-16:34:43.945] [ActivityExecutor-32] [INFO] [dku.datasets.file] running compute_Python27_NP - Building Filesystem handler config: {"path":"/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/uploads/SERVERCONFIGURATION/datasets/f_loan_data","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]}} [2020/12/17-16:34:43.946] [ActivityExecutor-32] [INFO] [dku.datasets.ftplike] running compute_Python27_NP - Enumerating Filesystem dataset prefix= [2020/12/17-16:34:43.946] [ActivityExecutor-32] [DEBUG] [dku.fs.local] running compute_Python27_NP - Enumerating local filesystem prefix=/ [2020/12/17-16:34:43.947] [ActivityExecutor-32] [DEBUG] [dku.fs.local] running compute_Python27_NP - Enumeration done nb_paths=1 size=1113566 [2020/12/17-16:34:43.947] [ActivityExecutor-32] [INFO] [dku.dataset.hash] running compute_Python27_NP - Caching readiness for dataset__cabhbwg__SERVERCONFIGURATION.f_loan_data__NP s=READY h=0kXinyc4XcT0QCON65UC9Q [2020/12/17-16:34:43.947] [ActivityExecutor-32] [DEBUG] [dku.flow.activity] running compute_Python27_NP - Recorded 1 hashes before activity run [2020/12/17-16:34:43.948] [ActivityExecutor-32] [DEBUG] [dku.flow.activity] running compute_Python27_NP - Building recipe runner of type [2020/12/17-16:34:43.963] [ActivityExecutor-32] [DEBUG] [dku.job.activity] running compute_Python27_NP - Filling source sizes [2020/12/17-16:34:43.966] [ActivityExecutor-32] [INFO] [dku.datasets.file] running compute_Python27_NP - Building Filesystem handler config: {"path":"/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/uploads/SERVERCONFIGURATION/datasets/f_loan_data","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]}} [2020/12/17-16:34:43.967] [ActivityExecutor-32] [INFO] [dku.datasets.ftplike] running compute_Python27_NP - Enumerating Filesystem dataset prefix= [2020/12/17-16:34:43.967] [ActivityExecutor-32] [DEBUG] [dku.fs.local] running compute_Python27_NP - Enumerating local filesystem prefix=/ [2020/12/17-16:34:43.968] [ActivityExecutor-32] [DEBUG] [dku.fs.local] running compute_Python27_NP - Enumeration done nb_paths=1 size=1113566 [2020/12/17-16:34:43.968] [ActivityExecutor-32] [DEBUG] [dku.job.activity] running compute_Python27_NP - Done filling source sizes [2020/12/17-16:34:43.969] [ActivityExecutor-32] [DEBUG] [dku.flow.activity] running compute_Python27_NP - Recipe runner built, will use 1 thread(s) [2020/12/17-16:34:43.969] [ActivityExecutor-32] [DEBUG] [dku.flow.activity] running compute_Python27_NP - Starting execution thread: com.dataiku.dip.recipes.code.python.PythonRecipeRunner@3c055189 [2020/12/17-16:34:43.969] [ActivityExecutor-32] [DEBUG] [dku.flow.activity] running compute_Python27_NP - Execution threads started, waiting for activity end [2020/12/17-16:34:43.972] [FRT-41-FlowRunnable] [INFO] [dku.flow.activity] act.compute_Python27_NP - Run thread for activity compute_Python27_NP starting [2020/12/17-16:34:43.973] [FRT-41-FlowRunnable] [INFO] [dku.flow.python] act.compute_Python27_NP - Starting execution of user's Python code [2020/12/17-16:34:43.979] [FRT-41-FlowRunnable] [INFO] [dku.venv.selector] act.compute_Python27_NP - Select code env lang=PYTHON projectSelection={"mode":"INHERIT","preventOverride":false} globalDefault=null [2020/12/17-16:34:43.980] [FRT-41-FlowRunnable] [INFO] [dku.flow.abstract.python] act.compute_Python27_NP - Dumping Python script to /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/script.py [2020/12/17-16:34:43.992] [FRT-41-FlowRunnable] [INFO] [dku.datasets.file] act.compute_Python27_NP - Building Filesystem handler config: {"path":"/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/uploads/SERVERCONFIGURATION/datasets/f_loan_data","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]}} [2020/12/17-16:34:43.996] [FRT-41-FlowRunnable] [INFO] [dku.datasets.file] act.compute_Python27_NP - Building Filesystem handler config: {"path":"/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/uploads/SERVERCONFIGURATION/datasets/f_loan_data","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]}} [2020/12/17-16:34:44.001] [FRT-41-FlowRunnable] [WARN] [dku.code.projectLibs] act.compute_Python27_NP - External libraries file not found: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/localconfig/projects/SERVERCONFIGURATION/lib/external-libraries.json [2020/12/17-16:34:44.003] [FRT-41-FlowRunnable] [INFO] [dku.code.projectLibs] act.compute_Python27_NP - EXTERNAL LIBS FROM SERVERCONFIGURATION is {"gitReferences":{},"pythonPath":["python"],"rsrcPath":["R"],"importLibrariesFromProjects":[]} [2020/12/17-16:34:44.003] [FRT-41-FlowRunnable] [INFO] [dku.code.projectLibs] act.compute_Python27_NP - chunkFolder is /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/localconfig/projects/SERVERCONFIGURATION/lib/R [2020/12/17-16:34:44.004] [FRT-41-FlowRunnable] [INFO] [xxx] act.compute_Python27_NP - RSRC PATH: [] [2020/12/17-16:34:44.005] [FRT-41-FlowRunnable] [INFO] [dku.recipes.code.base] act.compute_Python27_NP - Writing dku-exec-env for local execution in /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/remote-run-env-def.json [2020/12/17-16:34:44.011] [FRT-41-FlowRunnable] [INFO] [dku.code.envs.resolution] act.compute_Python27_NP - Executing Python activity in env: Python36 [2020/12/17-16:34:44.017] [FRT-41-FlowRunnable] [INFO] [dku.flow.abstract.python] act.compute_Python27_NP - Execute activity command: ["/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/bin/python","-u","/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/python-exec-wrapper.py","/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/script.py"] [2020/12/17-16:34:44.017] [FRT-41-FlowRunnable] [INFO] [dku.recipes.code.base] act.compute_Python27_NP - Run command insecurely, from user cabhbwg [2020/12/17-16:34:44.023] [FRT-41-FlowRunnable] [INFO] [dku.security.process] act.compute_Python27_NP - Setting exclusive access on /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8 [2020/12/17-16:34:44.024] [FRT-41-FlowRunnable] [DEBUG] [dku.security.fsacl] act.compute_Python27_NP - exec chown: sudo -n /etc/dataiku-security/mRGAADOQ7LAqGqwf6wfJeh9k/execwrapper.sh acls chown --path /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8 --owner cabhbwg [2020/12/17-16:34:44.105] [null-err-43] [INFO] [dku.utils] - 2020-12-17 16:34:44,104 43759 INFO Reading security conf from /etc/dataiku-security/mRGAADOQ7LAqGqwf6wfJeh9k [2020/12/17-16:34:44.108] [null-err-43] [INFO] [dku.utils] - 2020-12-17 16:34:44,104 43759 INFO Allowed path: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8 in /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data [2020/12/17-16:34:44.109] [null-err-43] [INFO] [dku.utils] - 2020-12-17 16:34:44,105 43759 INFO Reading security conf from /etc/dataiku-security/mRGAADOQ7LAqGqwf6wfJeh9k [2020/12/17-16:34:44.117] [null-err-43] [INFO] [dku.utils] - 2020-12-17 16:34:44,117 43759 INFO Allowed group ids: [610330, 681020] [2020/12/17-16:34:44.144] [null-err-43] [INFO] [dku.utils] - 2020-12-17 16:34:44,144 43759 INFO User cabhbwg belongs to groups: [610330] [2020/12/17-16:34:44.145] [null-err-43] [INFO] [dku.utils] - 2020-12-17 16:34:44,144 43759 INFO User cabhbwg belongs to group 610330 (cabhbwg) -> allowed [2020/12/17-16:34:44.146] [null-err-43] [INFO] [dku.utils] - 2020-12-17 16:34:44,144 43759 INFO Executing chown -Rh: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8 -> cabhbwg [2020/12/17-16:34:44.159] [Thread-26] [INFO] [dku.utils] - Done waiting for return value, got 0 [2020/12/17-16:34:44.166] [FRT-41-FlowRunnable] [DEBUG] [dku.security.fsacl] act.compute_Python27_NP - exec setfacl: sudo -n /etc/dataiku-security/mRGAADOQ7LAqGqwf6wfJeh9k/execwrapper.sh acls setfacl --path /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8 --perm u::rwx --perm g::--- --perm o:--- --perm d:u:cabhbwg:rwx --perm u:cabhbwg:rwx --recursive [2020/12/17-16:34:44.246] [null-err-46] [INFO] [dku.utils] - 2020-12-17 16:34:44,245 43766 INFO Reading security conf from /etc/dataiku-security/mRGAADOQ7LAqGqwf6wfJeh9k [2020/12/17-16:34:44.247] [null-err-46] [INFO] [dku.utils] - 2020-12-17 16:34:44,246 43766 INFO Allowed path: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8 in /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data [2020/12/17-16:34:44.248] [null-err-46] [INFO] [dku.utils] - 2020-12-17 16:34:44,246 43766 INFO Executing: ['setfacl', '-RP', '-m', 'u::rwx,g::---,o:---,d:u:cabhbwg:rwx,u:cabhbwg:rwx', '/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8'] [2020/12/17-16:34:44.259] [Thread-29] [INFO] [dku.utils] - Done waiting for return value, got 0 [2020/12/17-16:34:44.269] [FRT-41-FlowRunnable] [INFO] [dku.security.process] act.compute_Python27_NP - Written config to /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/config.json [2020/12/17-16:34:44.271] [FRT-41-FlowRunnable] [INFO] [dku.security.process] act.compute_Python27_NP - Starting wrapper: ["sudo","-n","/etc/dataiku-security/mRGAADOQ7LAqGqwf6wfJeh9k/execwrapper.sh","execute","/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/config.json"] [2020/12/17-16:34:44.342] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,341 43770 INFO Reading security conf from /etc/dataiku-security/mRGAADOQ7LAqGqwf6wfJeh9k [2020/12/17-16:34:44.343] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,342 43770 INFO Allowed group ids: [610330, 681020] [2020/12/17-16:34:44.347] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,346 43770 INFO User cabhbwg belongs to groups: [610330] [2020/12/17-16:34:44.348] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,347 43770 INFO User cabhbwg belongs to group 610330 (cabhbwg) -> allowed [2020/12/17-16:34:44.348] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,348 43770 INFO Reading security conf from /etc/dataiku-security/mRGAADOQ7LAqGqwf6wfJeh9k [2020/12/17-16:34:44.349] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,349 43770 INFO Allowed path: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8 in /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data [2020/12/17-16:34:44.350] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,349 43770 INFO [BF] Setting up process environment [2020/12/17-16:34:44.351] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,351 43770 INFO [BF] Writing wrapper pipes and pid to wrapper stdout [2020/12/17-16:34:44.353] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,353 43770 INFO [Parent] Started process 43772 [2020/12/17-16:34:44.354] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,353 43772 INFO [Child] pgroup is set [2020/12/17-16:34:44.354] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,354 43772 INFO [Child] opening stdin pipe: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/.stdin.4645ko7n [2020/12/17-16:34:44.374] [FRT-41-FlowRunnable] [INFO] [dku.security.process] act.compute_Python27_NP - Reading pipes from wrapper [2020/12/17-16:34:44.375] [FRT-41-FlowRunnable] [INFO] [dku.security.process] act.compute_Python27_NP - Pipes read as bytes: 767 [2020/12/17-16:34:44.376] [FRT-41-FlowRunnable] [INFO] [dku.security.process] act.compute_Python27_NP - Pipes read as bytes: {"pipes": {"stdin": "/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/.stdin.4645ko7n", "stderr": "/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/.stderr.4645ko7n", "stdout": "/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/.stdout.4645ko7n"}, "pid": 43770, "childPidFile": "/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/child.pid"} [2020/12/17-16:34:44.377] [FRT-41-FlowRunnable] [INFO] [dku.security.process] act.compute_Python27_NP - Pid read: 43770 [2020/12/17-16:34:44.378] [FRT-41-FlowRunnable] [INFO] [dku.security.process] act.compute_Python27_NP - Pipes read: { "stdin": "/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/.stdin.4645ko7n", "stderr": "/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/.stderr.4645ko7n", "stdout": "/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/.stdout.4645ko7n" } [2020/12/17-16:34:44.379] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,379 43772 INFO [Child] opened stdin pipe [2020/12/17-16:34:44.382] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,379 43772 INFO [Child] opening stdout pipe: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/.stdout.4645ko7n [2020/12/17-16:34:44.381] [FRT-41-FlowRunnable] [INFO] [dku.recipes.code.base] act.compute_Python27_NP - Process reads from nothing [2020/12/17-16:34:44.383] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,379 43772 INFO [Child] opened stdout [2020/12/17-16:34:44.384] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:44,379 43772 INFO [Child] opening stderr pipe: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/.stderr.4645ko7n [2020/12/17-16:34:44.387] [FRT-41-FlowRunnable] [INFO] [dku.resourceusage] act.compute_Python27_NP - Reporting start of CRU:{"context":{"type":"JOB_ACTIVITY","authIdentifier":"cabhbwg","projectKey":"SERVERCONFIGURATION","jobId":"Build_Python27_2020-12-17T15-34-42.420","activityId":"compute_Python27_NP","activityType":"recipe","recipeType":"python","recipeName":"compute_Python27"},"type":"LOCAL_PROCESS","id":"lAkO0RcLQSNv2aaZ","startTime":1608219284384,"localProcess":{"cpuCurrent":0.0}} [2020/12/17-16:34:44.382] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,379 43772 INFO [Child] opened stderr [2020/12/17-16:34:44.387] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,379 43772 INFO [Child] about to close other fd [2020/12/17-16:34:44.388] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,379 43772 INFO [Child] closed other fd [2020/12/17-16:34:44.388] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,380 43772 INFO [Child] chdired [2020/12/17-16:34:44.388] [FRT-41-FlowRunnable] [INFO] [dku.usage.computeresource.jek] act.compute_Python27_NP - Reporting start of resource usage: {"context":{"type":"JOB_ACTIVITY","authIdentifier":"cabhbwg","projectKey":"SERVERCONFIGURATION","jobId":"Build_Python27_2020-12-17T15-34-42.420","activityId":"compute_Python27_NP","activityType":"recipe","recipeType":"python","recipeName":"compute_Python27"},"type":"LOCAL_PROCESS","id":"lAkO0RcLQSNv2aaZ","startTime":1608219284384,"localProcess":{"cpuCurrent":0.0}} [2020/12/17-16:34:44.388] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,380 43772 INFO setting username=cabhbwg uid=610330 gid=610330 [2020/12/17-16:34:44.389] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,380 43772 INFO [Child] dropped privileges [2020/12/17-16:34:44.390] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,380 43772 INFO [Child] Checking access to DKUINSTALLDIR and DIP_HOME directories [2020/12/17-16:34:44.390] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,381 43772 INFO [Child] Executing: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/bin/python : /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/bin/python -u /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/python-exec-wrapper.py /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/script.py [2020/12/17-16:34:44.392] [process-resource-monitor-unkpid-52] [INFO] [dku.security.process] - GET PID CPF=/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/child.pid [2020/12/17-16:34:44.394] [process-resource-monitor-43772-52] [INFO] [dku.usage.computeresource.jek] - Reporting update of resource usage: {"context":{"type":"JOB_ACTIVITY","authIdentifier":"cabhbwg","projectKey":"SERVERCONFIGURATION","jobId":"Build_Python27_2020-12-17T15-34-42.420","activityId":"compute_Python27_NP","activityType":"recipe","recipeType":"python","recipeName":"compute_Python27"},"type":"LOCAL_PROCESS","id":"lAkO0RcLQSNv2aaZ","startTime":1608219284384,"localProcess":{"pid":43772,"commandName":"/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":120,"vmRSSMB":3,"vmHWMMB":3,"vmRSSAnonMB":1,"vmDataMB":1,"vmSizePeakMB":120,"vmRSSPeakMB":3,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}} [2020/12/17-16:34:44.394] [process-resource-monitor-43772-52] [DEBUG] [dku.resource] - Process stats for pid 43772: {"pid":43772,"commandName":"/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":120,"vmRSSMB":3,"vmHWMMB":3,"vmRSSAnonMB":1,"vmDataMB":1,"vmSizePeakMB":120,"vmRSSPeakMB":3,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0} [2020/12/17-16:34:44.430] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,430 INFO -------------------- [2020/12/17-16:34:44.431] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,430 INFO Dataiku Python entrypoint starting up [2020/12/17-16:34:44.432] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,430 INFO executable = /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/bin/python [2020/12/17-16:34:44.432] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,430 INFO argv = ['/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/python-exec-wrapper.py', '/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/script.py'] [2020/12/17-16:34:44.433] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,430 INFO -------------------- [2020/12/17-16:34:44.433] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,430 INFO Looking for RemoteRunEnvDef in ./remote-run-env-def.json [2020/12/17-16:34:44.434] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,430 INFO Found RemoteRunEnvDef environment: ./remote-run-env-def.json [2020/12/17-16:34:44.434] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,431 INFO Running a DSS Python recipe locally, uinsetting env [2020/12/17-16:34:44.434] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,431 INFO Setup complete, ready to execute Python code [2020/12/17-16:34:44.435] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,431 INFO Sys path: ['/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8', '/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/lib/python', '/GTU/BHBW/INSTANCE_DESIGN/apps/dss_cfg/dataiku-dss-8.0.2/python', '/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/lib64/python36.zip', '/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/lib64/python3.6', '/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/lib64/python3.6/lib-dynload', '/opt/rh/rh-python36/root/usr/lib64/python3.6', '/opt/rh/rh-python36/root/usr/lib/python3.6', '/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/lib/python3.6/site-packages'] [2020/12/17-16:34:44.435] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:44,431 INFO Script file: /GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/jobs/SERVERCONFIGURATION/Build_Python27_2020-12-17T15-34-42.420/compute_Python27_NP/python-recipe/pyouthtPjTodeIiv8/script.py [2020/12/17-16:34:45.481] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,480 INFO Initializing dataset writer for dataset SERVERCONFIGURATION.Python27 [2020/12/17-16:34:45.483] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,480 INFO Initializing write session [2020/12/17-16:34:45.505] [qtp593308118-26] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWritingService] - Init write session: HRchTTczOH [2020/12/17-16:34:45.506] [qtp593308118-26] [DEBUG] [dku.jobs] - Command /tintercom/datasets/init-write-session processed in 22ms [2020/12/17-16:34:45.511] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,510 INFO Starting RemoteStreamWriter [2020/12/17-16:34:45.512] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,511 INFO Initializing write data stream (HRchTTczOH) [2020/12/17-16:34:45.514] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,514 INFO Remote Stream Writer: start generate [2020/12/17-16:34:45.514] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,514 INFO Waiting for data to send ... [2020/12/17-16:34:45.523] [qtp593308118-27] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWriter] - Overwrite: clearing output [2020/12/17-16:34:45.523] [qtp593308118-27] [DEBUG] [dku.datasets.hdfs] - HDFS dataset handler dataset=SERVERCONFIGURATION.Python27 connection=hdfs_sentry cpr=/sentry/bhbwg/design resolvedPath=/SERVERCONFIGURATION/Python27 connRootSA=nullconnRootWithinSA=/sentry/bhbwg/design configuredRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 effectiveRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:45.525] [qtp593308118-27] [DEBUG] [dku.fsproviders.hdfs] - Build HDFSProvider conn=hdfs_sentry cpr=/sentry/bhbwg/design pWCR=/SERVERCONFIGURATION/Python27 crSA=null crWSA=/sentry/bhbwg/design rpWSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:45.525] [qtp593308118-27] [DEBUG] [dku.hadoop] - Initializing Hadoop FS with context UGI: cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS) (login: cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)) rootPathURI=null [2020/12/17-16:34:45.555] [qtp593308118-27] [INFO] [dku.hadoop] - Forcing disable of defaultScheme cache (#8388): hdfs [2020/12/17-16:34:45.560] [qtp593308118-27] [DEBUG] [dku.fsproviders.hdfs] - Built Hadoop FS for: null -> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-697784482_27, ugi=cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)]] [2020/12/17-16:34:45.697] [qtp593308118-27] [INFO] [dku.datasets.hdfs.permissions] - Setting dataset-wide data-usability ACL: SERVERCONFIGURATION.Python27 [2020/12/17-16:34:45.699] [qtp593308118-27] [INFO] [dku.fsproviders.hdfs] - ACL synchronization disabled, noop [2020/12/17-16:34:45.716] [qtp593308118-27] [DEBUG] [dku.hadoop] - Initializing Hadoop FS with context UGI: cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS) (login: cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)) rootPathURI=null [2020/12/17-16:34:45.745] [qtp593308118-27] [INFO] [dku.hadoop] - Forcing disable of defaultScheme cache (#8388): hdfs [2020/12/17-16:34:45.747] [qtp593308118-27] [INFO] [dku.fsproviders.hdfs] - delete recursive / -> /sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:45.766] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,765 INFO Sending data (1221746) [2020/12/17-16:34:45.767] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,766 INFO Waiting for data to send ... [2020/12/17-16:34:45.767] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,766 INFO Remote Stream Writer closed [2020/12/17-16:34:45.767] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:45,766 INFO Got end mark, ending send [2020/12/17-16:34:45.816] [qtp593308118-27] [INFO] [dku.fsproviders.hdfs] - is a directory true [2020/12/17-16:34:45.915] [qtp593308118-27] [INFO] [dku.fsproviders.hdfs] - after delete File does not exist: /sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:45.916] [qtp593308118-27] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWriter] - Creating output writer [2020/12/17-16:34:45.924] [qtp593308118-27] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWriter] - Initializing output writer [2020/12/17-16:34:45.930] [qtp593308118-27] [WARN] [dku.fsproviders.hdfs] - Dataset root path does not exist : /sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:45.930] [qtp593308118-27] [INFO] [dku.output.file] - Writing base=/ split=0 chunk=0 -> target = out-s0.csv [2020/12/17-16:34:45.984] [qtp593308118-27] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWriter] - Done initializing output writer [2020/12/17-16:34:46.103] [qtp593308118-27] [INFO] [dku.datasets] - Emitted 10000 rows [2020/12/17-16:34:46.157] [qtp593308118-27] [INFO] [dku.datasets] - Emitted 20000 rows [2020/12/17-16:34:46.423] [qtp593308118-27] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWritingService] - Pushed data to write session HRchTTczOH : 29092 rows [2020/12/17-16:34:46.424] [qtp593308118-23] [INFO] [com.dataiku.dip.dataflow.streaming.DatasetWritingService] - Finished write session: HRchTTczOH [2020/12/17-16:34:46.424] [qtp593308118-27] [DEBUG] [dku.jobs] - Command /tintercom/datasets/push-data processed in 907ms [2020/12/17-16:34:46.425] [qtp593308118-23] [DEBUG] [dku.jobs] - Command /tintercom/datasets/wait-write-session processed in 909ms [2020/12/17-16:34:46.430] [null-out-49] [INFO] [dku.utils] - 29092 rows successfully written (HRchTTczOH) [2020/12/17-16:34:46.430] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:46,430 INFO Check if spark is available [2020/12/17-16:34:46.433] [null-err-50] [INFO] [dku.utils] - 2020-12-17 16:34:46,431 INFO Not stopping a spark context: No module named 'pyspark' [2020/12/17-16:34:46.569] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:46,569 43770 INFO [Child] Process 43772 exited with exit=0 signal=0 [2020/12/17-16:34:46.570] [wrapper-stderr-48] [INFO] [dku.utils] - 2020-12-17 16:34:46,569 43770 INFO Full child code: 0 [2020/12/17-16:34:46.584] [FRT-41-FlowRunnable] [WARN] [dku.resource] act.compute_Python27_NP - stat file for pid 43772 does not exist. Process died? [2020/12/17-16:34:46.585] [FRT-41-FlowRunnable] [INFO] [dku.resourceusage] act.compute_Python27_NP - Reporting completion of CRU:{"context":{"type":"JOB_ACTIVITY","authIdentifier":"cabhbwg","projectKey":"SERVERCONFIGURATION","jobId":"Build_Python27_2020-12-17T15-34-42.420","activityId":"compute_Python27_NP","activityType":"recipe","recipeType":"python","recipeName":"compute_Python27"},"type":"LOCAL_PROCESS","id":"lAkO0RcLQSNv2aaZ","startTime":1608219284384,"localProcess":{"pid":43772,"commandName":"/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":120,"vmRSSMB":3,"vmHWMMB":3,"vmRSSAnonMB":1,"vmDataMB":1,"vmSizePeakMB":120,"vmRSSPeakMB":3,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}} [2020/12/17-16:34:46.585] [FRT-41-FlowRunnable] [INFO] [dku.usage.computeresource.jek] act.compute_Python27_NP - Reporting completion of resource usage: {"context":{"type":"JOB_ACTIVITY","authIdentifier":"cabhbwg","projectKey":"SERVERCONFIGURATION","jobId":"Build_Python27_2020-12-17T15-34-42.420","activityId":"compute_Python27_NP","activityType":"recipe","recipeType":"python","recipeName":"compute_Python27"},"type":"LOCAL_PROCESS","id":"lAkO0RcLQSNv2aaZ","startTime":1608219284384,"endTime":1608219286585,"localProcess":{"pid":43772,"commandName":"/GTU/BHBW/INSTANCE_DESIGN/apps/dss_data/code-envs/python/Python36/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":120,"vmRSSMB":3,"vmHWMMB":3,"vmRSSAnonMB":1,"vmDataMB":1,"vmSizePeakMB":120,"vmRSSPeakMB":3,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}} [2020/12/17-16:34:46.586] [FRT-41-FlowRunnable] [INFO] [dku.flow.python] act.compute_Python27_NP - Execution of user's Python code complete [2020/12/17-16:34:46.589] [FRT-41-FlowRunnable] [DEBUG] [dku.datasets.hdfs] act.compute_Python27_NP - HDFS dataset handler dataset=SERVERCONFIGURATION.Python27 connection=hdfs_sentry cpr=/sentry/bhbwg/design resolvedPath=/SERVERCONFIGURATION/Python27 connRootSA=nullconnRootWithinSA=/sentry/bhbwg/design configuredRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 effectiveRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:46.589] [FRT-41-FlowRunnable] [INFO] [dku.datasets.hdfs] act.compute_Python27_NP - Enumerating Filesystem dataset prefix= [2020/12/17-16:34:46.590] [FRT-41-FlowRunnable] [DEBUG] [dku.fsproviders.hdfs] act.compute_Python27_NP - Build HDFSProvider conn=hdfs_sentry cpr=/sentry/bhbwg/design pWCR=/SERVERCONFIGURATION/Python27 crSA=null crWSA=/sentry/bhbwg/design rpWSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:46.591] [FRT-41-FlowRunnable] [DEBUG] [dku.hadoop] act.compute_Python27_NP - Initializing Hadoop FS with context UGI: cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS) (login: cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)) rootPathURI=null [2020/12/17-16:34:46.608] [FRT-41-FlowRunnable] [INFO] [dku.hadoop] act.compute_Python27_NP - Forcing disable of defaultScheme cache (#8388): hdfs [2020/12/17-16:34:46.611] [FRT-41-FlowRunnable] [DEBUG] [dku.fsproviders.hdfs] act.compute_Python27_NP - Built Hadoop FS for: null -> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_962018015_41, ugi=cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)]] [2020/12/17-16:34:46.667] [FRT-41-FlowRunnable] [INFO] [dku.fsproviders.hdfs] act.compute_Python27_NP - Enumerating HDFS Filesystem from root : /sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:46.700] [FRT-41-FlowRunnable] [INFO] [dku.fsproviders.hdfs] act.compute_Python27_NP - HDFS-enumerate depth=0 curContent=1 [2020/12/17-16:34:46.702] [FRT-41-FlowRunnable] [INFO] [dku.fsproviders.hdfs] act.compute_Python27_NP - Done HDFS enumeration: nb_paths=1 total_size=1221746 [2020/12/17-16:34:46.707] [FRT-41-FlowRunnable] [INFO] [dku.flow.activity] act.compute_Python27_NP - Run thread done for activity compute_Python27_NP [2020/12/17-16:34:46.775] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - activity is finished [2020/12/17-16:34:46.777] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Executing default post-activity lifecycle hook [2020/12/17-16:34:46.783] [ActivityExecutor-32] [DEBUG] [dku.datasets.hdfs] running compute_Python27_NP - HDFS dataset handler dataset=SERVERCONFIGURATION.Python27 connection=hdfs_sentry cpr=/sentry/bhbwg/design resolvedPath=/SERVERCONFIGURATION/Python27 connRootSA=nullconnRootWithinSA=/sentry/bhbwg/design configuredRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 effectiveRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:46.789] [ActivityExecutor-32] [DEBUG] [dku.fsproviders.hdfs] running compute_Python27_NP - Build HDFSProvider conn=hdfs_sentry cpr=/sentry/bhbwg/design pWCR=/SERVERCONFIGURATION/Python27 crSA=null crWSA=/sentry/bhbwg/design rpWSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:46.789] [ActivityExecutor-32] [DEBUG] [dku.hadoop] running compute_Python27_NP - Initializing Hadoop FS with context UGI: cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS) (login: cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)) rootPathURI=null [2020/12/17-16:34:46.807] [ActivityExecutor-32] [INFO] [dku.hadoop] running compute_Python27_NP - Forcing disable of defaultScheme cache (#8388): hdfs [2020/12/17-16:34:46.809] [ActivityExecutor-32] [DEBUG] [dku.fsproviders.hdfs] running compute_Python27_NP - Built Hadoop FS for: null -> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-848738173_32, ugi=cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)]] [2020/12/17-16:34:46.872] [ActivityExecutor-32] [INFO] [dku.datasets.hdfs.permissions] running compute_Python27_NP - Setting dataset-wide data-usability ACL: SERVERCONFIGURATION.Python27 [2020/12/17-16:34:46.873] [ActivityExecutor-32] [INFO] [dku.fsproviders.hdfs] running compute_Python27_NP - ACL synchronization disabled, noop [2020/12/17-16:34:46.895] [ActivityExecutor-32] [DEBUG] [dku.datasets.hdfs] running compute_Python27_NP - HDFS dataset handler dataset=SERVERCONFIGURATION.Python27 connection=hdfs_sentry cpr=/sentry/bhbwg/design resolvedPath=/SERVERCONFIGURATION/Python27 connRootSA=nullconnRootWithinSA=/sentry/bhbwg/design configuredRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 effectiveRootPathWithinSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:46.895] [ActivityExecutor-32] [DEBUG] [dku.fsproviders.hdfs] running compute_Python27_NP - Build HDFSProvider conn=hdfs_sentry cpr=/sentry/bhbwg/design pWCR=/SERVERCONFIGURATION/Python27 crSA=null crWSA=/sentry/bhbwg/design rpWSA=/sentry/bhbwg/design/SERVERCONFIGURATION/Python27 [2020/12/17-16:34:46.896] [ActivityExecutor-32] [DEBUG] [dku.hadoop] running compute_Python27_NP - Initializing Hadoop FS with context UGI: cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS) (login: cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)) rootPathURI=null [2020/12/17-16:34:46.911] [ActivityExecutor-32] [INFO] [dku.hadoop] running compute_Python27_NP - Forcing disable of defaultScheme cache (#8388): hdfs [2020/12/17-16:34:46.914] [ActivityExecutor-32] [DEBUG] [dku.fsproviders.hdfs] running compute_Python27_NP - Built Hadoop FS for: null -> DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1310841563_32, ugi=cabhbwg (auth:PROXY) via cabhbwg/BDWCLUINT@INTBDA.BIL.COM (auth:KERBEROS)]] [2020/12/17-16:34:46.986] [ActivityExecutor-32] [INFO] [dku.datasets.hdfs.permissions] running compute_Python27_NP - Setting gateway ACL (create): SERVERCONFIGURATION.Python27 [2020/12/17-16:34:46.986] [ActivityExecutor-32] [INFO] [dku.fsproviders.hdfs] running compute_Python27_NP - ACL synchronization disabled, noop [2020/12/17-16:34:46.987] [ActivityExecutor-32] [INFO] [dku.datasets.hdfs.permissions] running compute_Python27_NP - Setting partition-wide data-usability ACL: SERVERCONFIGURATION.Python27partition= [2020/12/17-16:34:46.987] [ActivityExecutor-32] [INFO] [dku.fsproviders.hdfs] running compute_Python27_NP - ACL synchronization disabled, noop [2020/12/17-16:34:46.997] [ActivityExecutor-32] [INFO] [x] running compute_Python27_NP - **** Hadoop flavor **** {"flavor":"cdh5","hiveSupportsMREngine":true,"hive3":false} [2020/12/17-16:34:46.998] [ActivityExecutor-32] [INFO] [dku.hive.hiveserver2.pool] running compute_Python27_NP - take key clusterId=__builtin__ hadoopUser=cabhbwg db=d_bhbwg_design_serverconfiguration hiveConf=[]/> active=0 idle=0 [2020/12/17-16:34:46.999] [ActivityExecutor-32] [INFO] [dku.hive.hiveserver2.pool] running compute_Python27_NP - Creating a new HiveServer2 connection (db=d_bhbwg_design_serverconfiguration) [2020/12/17-16:34:47.001] [ActivityExecutor-32] [INFO] [dku.hive.hiveserver2.pool] running compute_Python27_NP - Connecting to HiveServer2: jdbc:hive2://bdw1n10.bnet.luxds.net:10000/default;ssl=true;sslTrustStore=/opt/cloudera/security/jks/BDWCLUINT.truststore;principal=hive/bdw1n10.bnet.luxds.net@INTBDA.BIL.COM [2020/12/17-16:34:47.004] [ActivityExecutor-32] [INFO] [dku.hive.hiveserver2.pool] running compute_Python27_NP - Driver version 1.1 [2020/12/17-16:34:47.016] [ActivityExecutor-32] [INFO] [org.apache.hive.jdbc.Utils] running compute_Python27_NP - Supplied authorities: bdw1n10.bnet.luxds.net:10000 [2020/12/17-16:34:47.016] [ActivityExecutor-32] [INFO] [org.apache.hive.jdbc.Utils] running compute_Python27_NP - Resolved authority: bdw1n10.bnet.luxds.net:10000 [2020/12/17-16:34:47.026] [ActivityExecutor-32] [WARN] [dku.flow.activity] running compute_Python27_NP - HIVE_METASTORE_SYNCHRONIZE: Failed to synchronize Hive metastore for dataset SERVERCONFIGURATION.Python27 com.dataiku.dip.exceptions.CodedException: in running compute_Python27_NP: Could not create HiveServer2 connection at com.dataiku.dip.hive.HiveServer2ConnectionPoolService.take(HiveServer2ConnectionPoolService.java:295) at com.dataiku.dip.hive.MetastoreInspectionService$HiveServer2MetastoreInspector.takeForMetastore(MetastoreInspectionService.java:166) at com.dataiku.dip.hive.MetastoreInspectionService$HiveServer2MetastoreInspector.takeForMetastore(MetastoreInspectionService.java:148) at com.dataiku.dip.hive.MetastoreInspectionService$HiveServer2MetastoreInspector.listHiveDatabase(MetastoreInspectionService.java:172) at com.dataiku.dip.hive.MetastoreInspectionService$HiveServer2MetastoreInspector.isHiveDatabase(MetastoreInspectionService.java:208) at com.dataiku.dip.hive.HiveServer2HiveMetastoreSynchronizer.isTableWriteSafe(HiveServer2HiveMetastoreSynchronizer.java:387) at com.dataiku.dip.hive.HiveServer2HiveMetastoreSynchronizer.synchronizeOneDatasetPartition(HiveServer2HiveMetastoreSynchronizer.java:298) at com.dataiku.dip.dataflow.jobrunner.ActivityRunner.waitForEnd(ActivityRunner.java:194) at com.dataiku.dip.dataflow.jobrunner.ActivityRunner.runActivity(ActivityRunner.java:600) at com.dataiku.dip.dataflow.jobrunner.JobRunner.runActivity(JobRunner.java:154) at com.dataiku.dip.dataflow.jobrunner.JobRunner.access$900(JobRunner.java:43) at com.dataiku.dip.dataflow.jobrunner.JobRunner$ActivityExecutorThread.run(JobRunner.java:342) Caused by: java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://bdw1n10.bnet.luxds.net:10000/default;ssl=true;sslTrustStore=/opt/cloudera/security/jks/BDWCLUINT.truststore;principal=hive/bdw1n10.bnet.luxds.net@INTBDA.BIL.COM: Error creating the transport at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:231) at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:168) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$HiveServer2ConnectionFactory.makeObject(HiveServer2ConnectionPoolService.java:393) at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$HiveServer2ConnectionFactory.makeObject(HiveServer2ConnectionPoolService.java:319) at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220) at com.dataiku.dip.hive.HiveServer2ConnectionPoolService.take(HiveServer2ConnectionPoolService.java:287) ... 11 more Caused by: org.apache.thrift.transport.TTransportException: Error creating the transport at org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:214) at org.apache.thrift.transport.TSSLTransportFactory.getClientSocket(TSSLTransportFactory.java:172) at org.apache.hadoop.hive.common.auth.HiveAuthUtils.getSSLSocket(HiveAuthUtils.java:64) at org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:371) at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:396) at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:201) ... 17 more Caused by: java.io.IOException: Could not load file: /opt/cloudera/security/jks/BDWCLUINT.truststore at org.apache.thrift.transport.TSSLTransportFactory.getStoreAsStream(TSSLTransportFactory.java:255) at org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:189) ... 22 more [2020/12/17-16:34:47.027] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Activity successful, computing metrics and running checks [2020/12/17-16:34:47.275] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Activity checks produced 0 values [2020/12/17-16:34:47.277] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Activity is successful, computing timestamps to propagate [2020/12/17-16:34:47.285] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Activity is successful, propagating 3 timestamps [2020/12/17-16:34:47.285] [ActivityExecutor-32] [INFO] [dku.timestamps.sql] running compute_Python27_NP - Starting to propagate 3 hashes [2020/12/17-16:34:47.303] [ActivityExecutor-32] [DEBUG] [dku.connections.sql.provider] running compute_Python27_NP - Commit conn=dssdb-h2-flow_state-CTEDeuX [2020/12/17-16:34:47.305] [ActivityExecutor-32] [INFO] [dku.timestamps.sql] running compute_Python27_NP - Hashes propagated in 19 ms [2020/12/17-16:34:47.305] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Removing samples for SERVERCONFIGURATION.Python27 [2020/12/17-16:34:47.307] [ActivityExecutor-32] [INFO] [dku.flow.activity] running compute_Python27_NP - Done post-activity tasks [2020/12/17-16:34:47.749] [qtp593308118-27] [INFO] [dku.jobs] - Connects using Shared secret [2020/12/17-16:34:47.749] [qtp593308118-27] [DEBUG] [dku.jobs] - Received command : /pintercom/stop_session [2020/12/17-16:34:47.750] [qtp593308118-27] [DEBUG] [dku.jobs] - Command /pintercom/stop_session processed in 1ms