[2025/05/14-08:07:34.876] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.prediction] - [ct: 0] ****************************************** [2025/05/14-08:07:34.876] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.prediction] - [ct: 0] ** Start train session s1 [2025/05/14-08:07:34.877] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.prediction] - [ct: 1] ****************************************** [2025/05/14-08:07:34.882] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.shaker.data] T-cqxrqRei - [ct: 6] Need to compute sampleId before checking memory cache [2025/05/14-08:07:34.883] [FT-TrainWorkThread-YgFeaaPB-47936] [DEBUG] [dip.shaker.runner] T-cqxrqRei - [ct: 7] Script settings sampleMax=104857600 processedMax=-1 [2025/05/14-08:07:34.883] [FT-TrainWorkThread-YgFeaaPB-47936] [DEBUG] [dip.shaker.runner] T-cqxrqRei - [ct: 7] Processing with sampleMax=104857600 processedMax=524288000 [2025/05/14-08:07:34.884] [FT-TrainWorkThread-YgFeaaPB-47936] [DEBUG] [dip.shaker.runner] T-cqxrqRei - [ct: 8] Computed required sample id : fdf17058675122ac38e5aa1f5cce399f-NA-67d2d4a7bb9f68bd5a120c6956099c590--d751713988987e9331980363e24189ce [2025/05/14-08:07:34.885] [FT-TrainWorkThread-YgFeaaPB-47936] [DEBUG] [dku.shaker.cache] T-cqxrqRei - Shaker MemoryCache get on dataset TUT_TS_ANALYSIS.train key=ds=a921e2a041cba2d1acd1aad378406324--scr=a5b24fbb2aa241080b0c971aaa7a2735--samp=fdf17058675122ac38e5aa1f5cce399f-NA-67d2d4a7bb9f68bd5a120c6956099c590--d751713988987e9331980363e24189ce: hit [2025/05/14-08:07:34.885] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.shaker.schema] T-cqxrqRei - [ct: 9] Column Ticker meaning=Text fail=0 [2025/05/14-08:07:34.886] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.shaker.schema] T-cqxrqRei - [ct: 10] Column Date meaning=Date fail=0 [2025/05/14-08:07:34.886] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.shaker.schema] T-cqxrqRei - [ct: 10] Column Open meaning=DoubleMeaning fail=0 [2025/05/14-08:07:34.887] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.shaker.schema] T-cqxrqRei - [ct: 11] Column High meaning=DoubleMeaning fail=0 [2025/05/14-08:07:34.887] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.shaker.schema] T-cqxrqRei - [ct: 11] Column Low meaning=DoubleMeaning fail=0 [2025/05/14-08:07:34.887] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.shaker.schema] T-cqxrqRei - [ct: 11] Column Close meaning=DoubleMeaning fail=0 [2025/05/14-08:07:34.888] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.shaker.schema] T-cqxrqRei - [ct: 12] Column Adj_close meaning=DoubleMeaning fail=0 [2025/05/14-08:07:34.888] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.shaker.schema] T-cqxrqRei - [ct: 12] Column Volume meaning=LongMeaning fail=0 [2025/05/14-08:07:34.890] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.datasets.file] T-cqxrqRei - [ct: 14] Building Filesystem handler config: {"connection":"filesystem_managed","path":"/TUT_TS_ANALYSIS.train","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]},"variablesExpansionLoopConfig":{"enabled":false,"mode":"CREATE_VARIABLE_FOR_EACH_COLUMN","replacements":[]}} [2025/05/14-08:07:34.890] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.datasets.fsbased] T-cqxrqRei - [ct: 14] gss {"connection":"filesystem_managed","path":"/TUT_TS_ANALYSIS.train","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]},"variablesExpansionLoopConfig":{"enabled":false,"mode":"CREATE_VARIABLE_FOR_EACH_COLUMN","replacements":[]}} [2025/05/14-08:07:34.891] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.datasets.ftplike] T-cqxrqRei - Variable-expanded selection rules: {"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]} [2025/05/14-08:07:34.891] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.datasets.ftplike] T-cqxrqRei - Enumerating Filesystem dataset prefix= [2025/05/14-08:07:34.891] [FT-TrainWorkThread-YgFeaaPB-47936] [DEBUG] [dku.datasets.fsbased] T-cqxrqRei - [ct: 15] Building FS provider for dataset handler: TUT_TS_ANALYSIS.train [2025/05/14-08:07:34.892] [FT-TrainWorkThread-YgFeaaPB-47936] [DEBUG] [dku.datasets.fsbased] T-cqxrqRei - [ct: 16] FS Provider built [2025/05/14-08:07:34.892] [FT-TrainWorkThread-YgFeaaPB-47936] [DEBUG] [dku.fs.local] T-cqxrqRei - [ct: 16] Enumerating local filesystem prefix=/ [2025/05/14-08:07:34.893] [FT-TrainWorkThread-YgFeaaPB-47936] [DEBUG] [dku.fs.local] T-cqxrqRei - [ct: 17] Enumeration done nb_paths=1 size=51053 [2025/05/14-08:07:34.893] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.input.push] T-cqxrqRei - USTP: push selection.method=COLUMN_ORDERED records=100000 ratio=0.02 col=Date [2025/05/14-08:07:34.894] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.format] T-cqxrqRei - [ct: 18] Extractor run: limit={"maxBytes":-1,"maxRecords":-1,"samplingMethod":"COLUMN_ORDERED","samplingRatio":0.02,"samplingSeed":1337,"ordering":{"enabled":false,"rules":[]}} totalRecords=0 [2025/05/14-08:07:34.894] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.format] T-cqxrqRei - [ct: 18] Start compressed [GZIP] stream: /data/dataiku/datadir/managed_datasets/TUT_TS_ANALYSIS.train/out-s0.csv.gz / totalRecsBefore=0 [2025/05/14-08:07:34.896] [FT-TrainWorkThread-YgFeaaPB-47936] [WARN] [dip.ordered.sample] T-cqxrqRei - Ordering column "Date" is neither numeric or temporal but DATE [2025/05/14-08:07:34.907] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.format] T-cqxrqRei - [ct: 31] after stream totalComp=51053 totalUncomp=177966 totalRec=2298 [2025/05/14-08:07:34.908] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.format] T-cqxrqRei - [ct: 32] Extractor run done, totalCompressed=51053 totalRecords=2298 [2025/05/14-08:07:34.908] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dip.sorter.chunk] T-cqxrqRei - Spilling chunk. used=1275132 [2025/05/14-08:07:34.918] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dip.ordered.sample] T-cqxrqRei - Sorter received 2298 rows, needs to produce 100000 [2025/05/14-08:07:34.941] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dip.ordered.sample] T-cqxrqRei - Final output got 2298 rows [2025/05/14-08:07:34.944] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.splits] T-cqxrqRei - [ct: 68] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,streamAll=true,kfold=true,folds=5,splitBeforePrepare=true,ds=train,sel=(method=column-ordered,column=Date,ascending=false,records=100000),r=0.8,s=1337, instance id: ea89d8caf552f670823e062aec93acd7-0 [2025/05/14-08:07:34.946] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.ml.python] T-cqxrqRei - [ct: 70] Joining processing thread ... [2025/05/14-08:07:34.946] [MRT-47937] [INFO] [dku.analysis.ml.python] - Running a preprocessing set: pp1 in /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1 [2025/05/14-08:07:34.957] [MRT-47937] [INFO] [org.eclipse.jetty.util.ssl.SslContextFactory] - x509=X509@7e63b810(1,h=[localhost, dss-design-host-node-325f0076-space-5dcbed3a-dku, dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local],a=[/127.0.0.1, /10.0.131.243],w=[]) for Server@75827689[provider=null,keyStore=file:///data/dataiku/datadir/install-support/encrypted-rpc/certandkey.p12,trustStore=null] [2025/05/14-08:07:34.962] [MRT-47937] [INFO] [dku.block.link] - Started a socket on port 39939 [2025/05/14-08:07:34.963] [MRT-47937] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers] - New worker pool created: pool-6axq6lurmz74ilgg [2025/05/14-08:07:34.963] [MRT-47937] [INFO] [dku.ml.kernel] - Writing output of python-single-command-kernel to /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/train.log [2025/05/14-08:07:34.964] [MRT-47937] [DEBUG] [dku.remoterun.registry] - Registering container execution doctor-preprocessing-vxjysuz of type DOCTOR_PYTHON [2025/05/14-08:07:34.964] [MRT-47937] [INFO] [dip.tickets] - Creating API ticket for analysis-ml-TUT_TS_ANALYSIS-vxJysuz on behalf of 4131210050_rakha@pknstan.ac.id id=analysis-ml-TUT_TS_ANALYSIS-vxJysuz_Bsi37TLnYA5J [2025/05/14-08:07:34.965] [MRT-47937] [INFO] [dku.containerexec.imageshelper] - Finding out which image to use for requestedBase=dss-gis-alma8-exec-base-13.5.1:1.0.78 envLang=PYTHON envName=time_series envVersion=null [2025/05/14-08:07:34.965] [MRT-47937] [INFO] [dku.containerexec.imageshelper] - Will use dss-gis-alma8-exec-base-13.5.1-1.0.78:pyenv-time_series-r-2025-05-09-09-02-28-625 [2025/05/14-08:07:34.967] [MRT-47937] [INFO] [dku.containers.kubernetes] - Trying to create pod dataiku-exec-doctor-preprocessing-vxjysuz, try=1 previousTransientErrors=0 previousQuotaErrors=0 timeSinceStartMS=0 [2025/05/14-08:07:34.967] [MRT-47937] [DEBUG] [dku.containers.kubernetes] - Pod creation command: kubectl --namespace space-5dcbed3a-dku-compute create -f /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/secret.yaml -f /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/pod.yaml [2025/05/14-08:07:35.311] [Thread-38173] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2025/05/14-08:07:35.312] [MRT-47937] [INFO] [dku.containers.kubernetes] - Pod was successfully created [2025/05/14-08:07:35.312] [MRT-47937] [INFO] [dku.containers.kubernetes] - Starting to wait for logs to be available [2025/05/14-08:07:35.312] [MRT-47937] [INFO] [dku.containers.kubernetes] - Getting pod [2025/05/14-08:07:35.404] [Thread-38176] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2025/05/14-08:07:36.404] [MRT-47937] [INFO] [dku.containers.kubernetes] - Pod is now created, waiting for logs fetch to be successful [2025/05/14-08:07:36.404] [MRT-47937] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-vxjysuz [2025/05/14-08:07:36.500] [Thread-38179] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2025/05/14-08:07:36.500] [MRT-47937] [INFO] [dku.containers.kubernetes] - Get logs failed [2025/05/14-08:07:36.500] [MRT-47937] [INFO] [dku.containers.kubernetes] - Waiting for logs, time elapsed: 96 milliseconds, status changed to: Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-vxjysuz" is waiting to start: ContainerCreating [2025/05/14-08:07:36.501] [MRT-47937] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-vxjysuz" is waiting to start: ContainerCreating ' [2025/05/14-08:07:37.501] [MRT-47937] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-vxjysuz [2025/05/14-08:07:37.607] [Thread-38182] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2025/05/14-08:07:37.607] [MRT-47937] [INFO] [dku.containers.kubernetes] - Getting logs succeeded, but checking that pod is indeed running [2025/05/14-08:07:37.697] [Thread-38185] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2025/05/14-08:07:37.698] [MRT-47937] [INFO] [dku.process] - Starting process (regular) [2025/05/14-08:07:37.700] [MRT-47937] [INFO] [dku.process] - Process started with pid=560894 [2025/05/14-08:07:37.700] [MRT-47937] [DEBUG] [dku.resourceusage] - Reporting start of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"4131210050_rakha@pknstan.ac.id","projectKey":"TUT_TS_ANALYSIS","analysisId":"ViMgkniW","mlTaskId":"cqxrqRei","sessionId":"s1"},"type":"SINGLE_K8S_JOB","id":"HYbf04OLO5xrkld1","startTime":1747210054963,"singleK8SJob":{"k8sClusterId":"__builtin__","executionId":"doctor-preprocessing-vxjysuz","cpuRequest":1.0,"cpuLimit":1.0,"memoryRequestMB":4096,"memoryLimitMB":4096}} [2025/05/14-08:07:37.701] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.ml.kernel] - Waiting for K8S pod termination [2025/05/14-08:07:37.701] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.containers.kubernetes] - wait for pod to complete, getting pod status [2025/05/14-08:07:37.702] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-vxjysuz- try number 1 [2025/05/14-08:07:37.713] [MRT-47937] [INFO] [dku.link.secret_protected] - Connected to kernel [2025/05/14-08:07:37.716] [MRT-47937] [INFO] [dku.block.link.interaction] - Execute link command respClazz=true respTypeToken=false respIsString=false is=false asyncInputStream=false os=false [2025/05/14-08:07:37.817] [Thread-38191] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2025/05/14-08:07:37.817] [KNL-python-single-command-kernel-k8smonitor-47962] [DEBUG] [dku.containers.kubernetes] - kubectl get pod OK, out=Running [2025/05/14-08:07:37.818] [KNL-python-single-command-kernel-k8smonitor-47962] [DEBUG] [dku.containers.kubernetes] - Kubernetes pod is running /usr/lib64/python3.9/runpy.py:127: RuntimeWarning: 'dataiku.container.runner' found in sys.modules after import of package 'dataiku.container', but prior to execution of 'dataiku.container.runner'; this may result in unpredictable behaviour warn(RuntimeWarning(msg)) [2025-05-14 08:07:36,685] [1/MainThread] [INFO] [root] Fetching job definition [2025-05-14 08:07:36,685] [1/MainThread] [DEBUG] [root] Creating HTTP session for internal RPC [2025-05-14 08:07:36,685] [1/MainThread] [DEBUG] [root] Verifying SSL calls: /home/dataiku/rpc_server_cert.pem [2025-05-14 08:07:36,687] [1/MainThread] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 [2025-05-14 08:07:36,710] [1/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/get-execution HTTP/1.1" 200 None [2025-05-14 08:07:36,710] [1/MainThread] [INFO] [root] got exec: {"dssVersion": "13.5.1", "id": "doctor-preprocessing-vxjysuz", "projectKey": "TUT_TS_ANALYSIS", "backendDipHomePath": "/data/dataiku/datadir", "contextPath": "/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei", "executionPath": "/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1", "resyncPythonCode": false, "type": "DOCTOR_PYTHON", "definition": null, "payload": null, "codeEnvs": [{"lang": "PYTHON", "name": "time_series", "version": "Latest"}], "envResource": null} [2025-05-14 08:07:36,710] [1/MainThread] [INFO] [root] Running with DSS code from image [2025-05-14 08:07:36,711] [1/MainThread] [INFO] [root] Training doctor model [2025-05-14 08:07:36,711] [1/MainThread] [INFO] [root] Fetching project libraries [2025-05-14 08:07:36,715] [1/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/get-file HTTP/1.1" 200 12288 [2025-05-14 08:07:36,718] [1/MainThread] [INFO] [root] Fetching instance libraries [2025-05-14 08:07:36,722] [1/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/get-file HTTP/1.1" 200 None [2025-05-14 08:07:36,731] [1/MainThread] [DEBUG] [root] Adding /home/dataiku/lib/project/project-python-libs/TUT_TS_ANALYSIS/python to Pythonpath [2025-05-14 08:07:36,731] [1/MainThread] [INFO] [root] running on python 3.9.20 (main, Oct 23 2024, 09:34:14) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] [2025-05-14 08:07:36,732] [1/MainThread] [INFO] [root] now using PYTHONPATH: /home/dataiku:/opt/dataiku/python:/home/dataiku/lib/instance:/home/dataiku/lib/project/project-python-libs/TUT_TS_ANALYSIS/python [2025-05-14 08:07:36,732] [1/MainThread] [INFO] [root] Running doctor server [2025-05-14 08:07:36,737] [1/MainThread] [INFO] [root] Running subprocess in PID 7 [2025-05-14 08:07:36,737] [1/Thread-1] [INFO] [root] Start monitoring process 7 [2025-05-14 08:07:37,454] [7/MainThread] [INFO] [root] Launching doctor command server [2025-05-14 08:07:37,454] [7/MainThread] [INFO] [dataiku.base.socket_block_link] cert found [2025-05-14 08:07:37,456] [7/MainThread] [INFO] [dataiku.base.socket_block_link] Connecting to dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local (10.0.131.243) at port 39939 [2025-05-14 08:07:37,713] [7/MainThread] [INFO] [dataiku.base.socket_block_link] Connected to dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local (10.0.131.243) at port 39939 /opt/dataiku/code-env/lib/python3.9/site-packages/jinja2/loaders.py:233: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html from pkg_resources import DefaultProvider [2025-05-14 08:07:38,145] [7/MainThread] [INFO] [root] Running analysis command: train_prediction_timeseries [2025-05-14 08:07:38,156] [7/MainThread] [WARNING] [dataiku.doctor.utils.gpu_execution] /bin/sh: nvidia-smi: command not found [2025-05-14 08:07:38,163] [7/MainThread] [WARNING] [dataiku.doctor.utils.gpu_execution] /bin/sh: nvidia-smi: command not found [2025-05-14 08:07:38,164] [7/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: DatasetSanityCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_DATASET_SANITY_CHECKS [2025-05-14 08:07:38,164] [7/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: ModelCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK [2025-05-14 08:07:38,164] [7/MainThread] [INFO] [dataiku.doctor.utils.listener] START - Loading source dataset [2025-05-14 08:07:38,164] [7/MainThread] [INFO] [dataiku.base.folder_context] Fetching file from backend: source='/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json' target='/home/dataiku/cache/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json' [2025-05-14 08:07:38,165] [7/MainThread] [INFO] [dataiku.container.file_exchange] Fetching file /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json into /home/dataiku/cache/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json [2025-05-14 08:07:38,166] [7/MainThread] [DEBUG] [root] Creating HTTP session for internal RPC [2025-05-14 08:07:38,166] [7/MainThread] [DEBUG] [root] Verifying SSL calls: /home/dataiku/rpc_server_cert.pem [2025-05-14 08:07:38,169] [7/MainThread] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 [2025-05-14 08:07:38,186] [7/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/get-file HTTP/1.1" 200 691 [2025-05-14 08:07:38,201] [7/MainThread] [INFO] [dataiku.doctor.utils] Training on platform: {"architecture": ["64bit", "ELF"], "libc": ["glibc", "2.28"], "machine": "x86_64", "kernelRelease": "6.1.119-129.201.amzn2023.x86_64", "kernelVersion": "#1 SMP PREEMPT_DYNAMIC Tue Dec 3 21:07:35 UTC 2024", "platform": "Linux-6.1.119-129.201.amzn2023.x86_64-x86_64-with-glibc2.28", "pythonCompiler": "GCC 8.5.0 20210514 (Red Hat 8.5.0-22)", "pythonImplementation": "CPython", "pythonVersion": "3.9.20", "system": "Linux"} [2025-05-14 08:07:38,202] [7/MainThread] [INFO] [dataiku.doctor.utils] Training on CPU: {"processor": "0", "vendor_id": "GenuineIntel", "cpu family": "6", "model": "85", "model name": "Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz", "stepping": "7", "microcode": "0x5003707", "cpu MHz": "3099.107", "cache size": "36608 KB", "physical id": "0", "siblings": "8", "core id": "0", "cpu cores": "4", "apicid": "0", "initial apicid": "0", "fpu": "yes", "fpu_exception": "yes", "cpuid level": "13", "wp": "yes", "flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke", "bugs": "cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data retbleed gds bhi", "bogomips": "4999.99", "clflush size": "64", "cache_alignment": "64", "address sizes": "46 bits physical, 48 bits virtual"} [2025-05-14 08:07:38,216] [7/MainThread] [INFO] [dataiku.doctor.utils] Training with main package versions: {"pandas": "1.3.5", "scikit-learn": "1.0.2", "scipy": "1.5.4", "lightgbm": "3.2.1", "gluonts": "0.9.10", "pmdarima": "1.8.4", "prophet": "1.1.3", "scikit-optimize": "0.9.0", "flask": "1.0.4", "itsdangerous": "2.0.1", "jinja2": "2.11.3", "xgboost": "0.82", "werkzeug": "2.1.2", "pydantic": "1.10.22", "statsmodels": "0.12.2", "pillow": "11.2.1", "mxnet": "1.8.0.post0", "markupsafe": "2.0.1", "cloudpickle": "1.5.0"} [2025-05-14 08:07:38,217] [7/MainThread] [INFO] [root] Sending /home/dataiku/cache/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json to /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json [2025-05-14 08:07:38,219] [7/MainThread] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local [2025-05-14 08:07:38,235] [7/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/put-file?executionId=doctor-preprocessing-vxjysuz&fileKind=FILTERED_PATHS&path=%2Fdata%2Fdataiku%2Fdatadir%2Fanalysis-data%2FTUT_TS_ANALYSIS%2FViMgkniW%2FcqxrqRei%2Fsessions%2Fs1%2Fpp1%2Fm1%2Ftrain_info.json&expand=false HTTP/1.1" 200 2 [2025-05-14 08:07:38,236] [7/MainThread] [INFO] [root] Sending /home/dataiku/cache/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json to /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json [2025-05-14 08:07:38,237] [7/MainThread] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local [2025-05-14 08:07:38,252] [7/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/put-file?executionId=doctor-preprocessing-vxjysuz&fileKind=FILTERED_PATHS&path=%2Fdata%2Fdataiku%2Fdatadir%2Fanalysis-data%2FTUT_TS_ANALYSIS%2FViMgkniW%2FcqxrqRei%2Fsessions%2Fs1%2Fpp1%2Fm1%2Ftrain_info.json&expand=false HTTP/1.1" 200 2 [2025-05-14 08:07:38,252] [7/MainThread] [INFO] [dataiku.base.folder_context] Fetching file from backend: source='/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/splits/full-ea89d8caf552f670823e062aec93acd7-0.csv.gz' target='/home/dataiku/cache/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/splits/full-ea89d8caf552f670823e062aec93acd7-0.csv.gz' [2025-05-14 08:07:38,252] [7/MainThread] [INFO] [dataiku.container.file_exchange] Fetching file /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/splits/full-ea89d8caf552f670823e062aec93acd7-0.csv.gz into /home/dataiku/cache/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/splits/full-ea89d8caf552f670823e062aec93acd7-0.csv.gz [2025-05-14 08:07:38,253] [7/MainThread] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local [2025-05-14 08:07:38,268] [7/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/get-file HTTP/1.1" 200 50474 [2025-05-14 08:07:38,269] [7/MainThread] [INFO] [root] Reading with dtypes: None [2025-05-14 08:07:38,269] [7/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Ticker: (schema_type=string feature_type=CATEGORY feature_role=TIMESERIES_IDENTIFIER) [2025-05-14 08:07:38,269] [7/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Date: None (schema_type=date feature_type=NUMERIC feature_role=TIME) [2025-05-14 08:07:38,269] [7/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Open: None (schema_type=double feature_type=NUMERIC feature_role=REJECT) [2025-05-14 08:07:38,269] [7/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for High: None (schema_type=double feature_type=NUMERIC feature_role=REJECT) [2025-05-14 08:07:38,269] [7/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Low: None (schema_type=double feature_type=NUMERIC feature_role=REJECT) [2025-05-14 08:07:38,269] [7/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Close: None (schema_type=double feature_type=NUMERIC feature_role=REJECT) [2025-05-14 08:07:38,269] [7/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Adj_close: (schema_type=double feature_type=NUMERIC feature_role=TARGET) [2025-05-14 08:07:38,269] [7/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Volume: None (schema_type=bigint feature_type=NUMERIC feature_role=REJECT) [2025-05-14 08:07:38,270] [7/MainThread] [INFO] [root] Reading with FIXED dtypes: {'Ticker': , 'Adj_close': } [2025-05-14 08:07:38,291] [7/MainThread] [INFO] [root] Loaded table [2025-05-14 08:07:38,292] [7/MainThread] [INFO] [dataiku.doctor.utils] Normalizing date to datetime and converting to UTC : 0 2021-12-27 00:00:00+00:00 1 2021-12-27 00:00:00+00:00 2 2021-12-27 00:00:00+00:00 3 2021-12-20 00:00:00+00:00 4 2021-12-20 00:00:00+00:00 ... 2293 2007-05-07 00:00:00+00:00 2294 2007-05-07 00:00:00+00:00 2295 2007-04-30 00:00:00+00:00 2296 2007-04-30 00:00:00+00:00 2297 2007-04-30 00:00:00+00:00 Name: Date, Length: 2298, dtype: datetime64[ns, UTC] [2025-05-14 08:07:38,297] [7/MainThread] [INFO] [dataiku.doctor.utils] Normalized date : 0 2021-12-27 1 2021-12-27 2 2021-12-27 3 2021-12-20 4 2021-12-20 ... 2293 2007-05-07 2294 2007-05-07 2295 2007-04-30 2296 2007-04-30 2297 2007-04-30 Name: Date, Length: 2298, dtype: datetime64[ns] [2025-05-14 08:07:38,297] [7/MainThread] [INFO] [dataiku.doctor.utils] Coercion done [2025-05-14 08:07:38,297] [7/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded train df: shape=(2298,8) [2025-05-14 08:07:38,297] [7/MainThread] [INFO] [dataiku.doctor.utils.listener] END - Loading source dataset [2025-05-14 08:07:38,297] [7/MainThread] [INFO] [dataiku.doctor.utils.listener] START - Time series resampling [2025-05-14 08:07:38,298] [7/MainThread] [INFO] [root] Sending /home/dataiku/cache/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json to /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json [2025-05-14 08:07:38,301] [7/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/put-file?executionId=doctor-preprocessing-vxjysuz&fileKind=FILTERED_PATHS&path=%2Fdata%2Fdataiku%2Fdatadir%2Fanalysis-data%2FTUT_TS_ANALYSIS%2FViMgkniW%2FcqxrqRei%2Fsessions%2Fs1%2Fpp1%2Fm1%2Ftrain_info.json&expand=false HTTP/1.1" 200 2 [2025-05-14 08:07:38,303] [7/MainThread] [INFO] [dataiku.doctor.timeseries.preparation.preprocessing] Resampling with params: {'numericalInterpolateMethod': 'LINEAR', 'numericalExtrapolateMethod': 'NO_EXTRAPOLATION', 'categoricalImputeMethod': 'PREVIOUS_NEXT', 'duplicateTimestampsHandlingMethod': 'FAIL_IF_CONFLICTING', 'startDateMode': 'AUTO', 'customStartDate': '2007-04-30', 'endDateMode': 'AUTO', 'customEndDate': '2021-12-27', 'numericalColumns': 1, 'categoricalColumns': 0, 'identifiersColumns': 1, 'numberOfTimeunits': 1, 'timeUnit': 'WEEK', 'endOfWeekDay': 'MON', 'unitAlignment': 0, 'monthlyAlignment': None} [2025-05-14 08:07:38,314] [7/MainThread] [INFO] [dataiku.doctor.timeseries.preparation.resampling] Found 3 time series to resample [2025-05-14 08:07:38,314] [7/MainThread] [INFO] [dataiku.doctor.timeseries.preparation.resampling] Resampling time series (Ticker: AAL) of shape: (766, 8) [2025-05-14 08:07:38,327] [7/MainThread] [INFO] [dataiku.doctor.timeseries.preparation.resampling] Resampling time series (Ticker: DAL) of shape: (766, 8) [2025-05-14 08:07:38,337] [7/MainThread] [INFO] [dataiku.doctor.timeseries.preparation.resampling] Resampling time series (Ticker: UAL) of shape: (766, 8) [2025-05-14 08:07:38,352] [7/MainThread] [INFO] [dataiku.doctor.utils.listener] END - Time series resampling [2025-05-14 08:07:38,352] [7/MainThread] [INFO] [dataiku.doctor.timeseries.train.launch_training] Time series split: using a fold offset for hyperparameters search and evaluation [2025-05-14 08:07:38,352] [7/MainThread] [INFO] [dataiku.doctor.utils.listener] START - Preparing splits [2025-05-14 08:07:38,353] [7/MainThread] [INFO] [root] Sending /home/dataiku/cache/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json to /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json [2025-05-14 08:07:38,356] [7/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/put-file?executionId=doctor-preprocessing-vxjysuz&fileKind=FILTERED_PATHS&path=%2Fdata%2Fdataiku%2Fdatadir%2Fanalysis-data%2FTUT_TS_ANALYSIS%2FViMgkniW%2FcqxrqRei%2Fsessions%2Fs1%2Fpp1%2Fm1%2Ftrain_info.json&expand=false HTTP/1.1" 200 2 [2025-05-14 08:07:38,356] [7/MainThread] [INFO] [dataiku.doctor.timeseries.models] Create CLF from params: {'algorithm': 'GLUONTS_NPTS_FORECASTER', 'gluonts_npts_timeseries_grid': {'context_length': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1}, 'values': [1100], 'range': {'min': 10, 'max': 20, 'nbValues': 3, 'scaling': 'LINEAR'}}, 'kernel_type': {'values': {'exponential': {'enabled': True}, 'uniform': {'enabled': False}}}, 'exp_kernel_weights': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1e-23}, 'values': [0.5, 1.0, 2.0], 'range': {'min': 0.1, 'max': 10.0, 'nbValues': 3, 'scaling': 'LINEAR'}}, 'feature_scale': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1e-23}, 'values': [1000.0], 'range': {'min': 0.1, 'max': 10000.0, 'nbValues': 3, 'scaling': 'LINEAR'}}, 'full_context': True, 'use_seasonal_model': True, 'use_default_time_features': True, 'seed': 1337, 'enabled': True}, 'max_ensemble_nodes_serialized': 0, 'metrics': {'customMetrics': [], 'evaluationMetric': 'MAPE', 'classAveragingMethod': 'MACRO', 'costMatrixWeights': {'tpGain': 1.0, 'tnGain': 0.0, 'fpGain': -0.3, 'fnGain': 0.0}, 'liftPoint': 0.4, 'netUpliftPoint': 0.5, 'causalWeighting': 'NO_WEIGHTING'}, 'autoOptimizeThreshold': True, 'forcedClassifierThreshold': 0.0, 'gridLength': 3, 'grid_search_params': {'mode': 'TIME_SERIES_KFOLD', 'splitRatio': 0.8, 'shuffleIterations': 1, 'nFolds': 3, 'cvSeed': 1337, 'stratified': True, 'grouped': False, 'strategy': 'GRID', 'bayesianOptimizer': 'SCIKIT_OPTIMIZE', 'randomized': True, 'seed': 1337, 'nIter': 0, 'nIterRandom': 3, 'timeout': 0, 'nJobs': 1, 'distributed': False, 'nContainers': 4, 'foldOffset': True, 'equalDurationFolds': False}, 'pluginAlgoCustomGridSearch': False, 'computeLearningCurves': False, 'skipExpensiveReports': False, 'propensityModeling': {'enabled': True, 'calibrateProbabilities': False, 'calibrationDataRatio': 0.1}} for algorithm GLUONTS_NPTS_FORECASTER /opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/compat/pandas.py:12: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. version = LooseVersion(pd.__version__) /opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/compat/pandas.py:14: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. pandas_lt_1_0_0 = version < LooseVersion('1.0.0') /opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/compat/pandas.py:15: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. pandas_lt_25_0 = version < LooseVersion('0.25.0') /opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/compat/pandas.py:16: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. pandas_gte_23_0 = version >= LooseVersion('0.23.0') /opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/compat/numpy.py:46: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. NP_LT_114 = LooseVersion(np.__version__) < LooseVersion('1.14') /opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/compat/scipy.py:6: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. SP_VERSION = LooseVersion(scipy.__version__) /opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/compat/scipy.py:7: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. SCIPY_11 = SP_VERSION < LooseVersion("1.2.0") /opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/compat/scipy.py:10: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. SCIPY_GT_14 = SP_VERSION >= LooseVersion("1.5") /opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/compat/scipy.py:11: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. SP_LT_15 = SP_VERSION < LooseVersion("1.5") [2025/05/14-08:07:38.369] [MRT-47937] [INFO] [dku.block.link.interaction] - Check result for nullity exceptionIfNull=true result=null [2025-05-14 08:07:38,369] [7/MainThread] [INFO] [dataiku.doctor.utils.listener] END - Preparing splits Traceback (most recent call last): File "/opt/dataiku/python/dataiku/doctor/server.py", line 45, in serve ret = api_command(arg) File "/opt/dataiku/python/dataiku/doctor/dkuapi.py", line 46, in aux return api(**kwargs) File "/opt/dataiku/python/dataiku/doctor/commands.py", line 979, in train_prediction_timeseries launch_training(core_params, modeling_sets, preprocessing_params, resampling_params, preprocessing_folder_context, split_folder_context, split_desc) File "/opt/dataiku/python/dataiku/doctor/timeseries/train/launch_training.py", line 102, in launch_training search_runner = algorithm.get_search_runner(search_split_handler, core_params, modeling_params, File "/opt/dataiku/python/dataiku/doctor/timeseries/models/__init__.py", line 60, in get_search_runner trainable_model = self.model_from_params(input_hp_space, modeling_params, core_params) File "/opt/dataiku/python/dataiku/doctor/timeseries/models/__init__.py", line 169, in model_from_params from dataiku.doctor.timeseries.models.gluonts.npts import DkuNPTSEstimator File "/opt/dataiku/python/dataiku/doctor/timeseries/models/gluonts/npts.py", line 2, in from dataiku.doctor.timeseries.models.gluonts.base_estimator import DkuGluonTSEstimator File "/opt/dataiku/python/dataiku/doctor/timeseries/models/gluonts/base_estimator.py", line 8, in from dataiku.doctor.timeseries.models.base_estimator import BaseTimeseriesEstimator File "/opt/dataiku/python/dataiku/doctor/timeseries/models/base_estimator.py", line 10, in from statsmodels.tsa.stattools import acf File "/opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/tsa/stattools.py", line 16, in from statsmodels.regression.linear_model import OLS, yule_walker File "/opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/regression/__init__.py", line 1, in from .linear_model import yule_walker File "/opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/regression/linear_model.py", line 46, in import statsmodels.base.model as base File "/opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/base/model.py", line 16, in from statsmodels.tools.numdiff import approx_fprime File "/opt/dataiku/code-env/lib64/python3.9/site-packages/statsmodels/tools/numdiff.py", line 51, in EPS = np.MachAr().eps File "/opt/dataiku/code-env/lib64/python3.9/site-packages/numpy/__init__.py", line 320, in __getattr__ raise AttributeError("module {!r} has no attribute " AttributeError: module 'numpy' has no attribute 'MachAr' [2025-05-14 08:07:38,372] [7/MainThread] [INFO] [dataiku.base.socket_block_link] Client closed Installing debugging signal handler [2025-05-14 08:07:38,594] [1/MainThread] [INFO] [root] Stop monitoring process 7 [2025/05/14-08:07:38.872] [MRT-47937] [INFO] [dku.kernels] - Getting kernel tail [2025/05/14-08:07:38.874] [MRT-47937] [INFO] [dku.ml.kernel] - Kubernetes: Trying to enrich exception: com.dataiku.dip.io.SocketBlockLinkKernelException: Failed to train : : module 'numpy' has no attribute 'MachAr' from kernel com.dataiku.dip.analysis.coreservices.AnalysisMLContainerKernel@74b89089 process=com.dataiku.dip.containers.exec.KubernetesExecUtils$KubernetesPod@2a44144e pid=560894 retcode=null [2025/05/14-08:07:38.874] [MRT-47937] [WARN] [dku.analysis.ml.python] - Training failed com.dataiku.dip.io.SocketBlockLinkKernelException: Failed to train : : module 'numpy' has no attribute 'MachAr' at com.dataiku.dip.io.SocketBlockLinkInteraction.throwExceptionFromPython(SocketBlockLinkInteraction.java:306) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.checkException(SocketBlockLinkInteraction.java:219) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.get(SocketBlockLinkInteraction.java:194) at com.dataiku.dip.io.SingleCommandKernelLink$1.call(SingleCommandKernelLink.java:215) at com.dataiku.dip.analysis.ml.prediction.PredictionTrainAdditionalThread.process(PredictionTrainAdditionalThread.java:76) at com.dataiku.dip.analysis.ml.shared.PRNSTrainThread.run(PRNSTrainThread.java:164) [2025/05/14-08:07:38.874] [MRT-47937] [INFO] [dku.ml.kernel] - Waiting for error after pod failed [2025/05/14-08:07:38.874] [MRT-47937] [INFO] [dku.ml.kernel] - /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/error.json not here, still waiting [2025/05/14-08:07:41.375] [MRT-47937] [INFO] [dku.ml.kernel] - /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/error.json not here, still waiting [2025-05-14 08:07:41,743] [1/Thread-1] [WARNING] [root] Process not yet started, or shutting down [2025-05-14 08:07:41,743] [1/Thread-1] [INFO] [root] Done monitoring process 7 [2025-05-14 08:07:41,743] [1/MainThread] [INFO] [root] Containerized process exited with status 0 [2025-05-14 08:07:41,747] [1/MainThread] [DEBUG] [urllib3.connectionpool] https://dss-headless-node-325f0076.space-5dcbed3a-dku.svc.cluster.local:10005 "POST /dip/api/tintercom/containers/put-file?executionId=doctor-preprocessing-vxjysuz&fileKind=EXECUTION_DIR&path=container_done.txt HTTP/1.1" 200 2 Installing debugging signal handler [2025/05/14-08:07:42.818] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.containers.kubernetes] - wait for pod to complete, getting pod status [2025/05/14-08:07:42.818] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-vxjysuz- try number 1 [2025/05/14-08:07:42.907] [Thread-38194] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2025/05/14-08:07:42.907] [KNL-python-single-command-kernel-k8smonitor-47962] [DEBUG] [dku.containers.kubernetes] - kubectl get pod OK, out=Running [2025/05/14-08:07:42.907] [KNL-python-single-command-kernel-k8smonitor-47962] [DEBUG] [dku.containers.kubernetes] - Kubernetes pod is running [2025/05/14-08:07:43.875] [MRT-47937] [INFO] [dku.ml.kernel] - Didn't get error file [2025/05/14-08:07:43.875] [MRT-47937] [INFO] [dku.ml.kernel] - Deleting pod [2025/05/14-08:07:43.876] [MRT-47937] [INFO] [dku.containers.kubernetes] - Deleting Kubernetes object: ["delete","-f","/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/pod.yaml","-f","/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/secret.yaml"] [2025/05/14-08:07:43.991] [Thread-38197] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2025/05/14-08:07:43.992] [MRT-47937] [INFO] [dku.ml.kernel] - Pod deleted [2025/05/14-08:07:43.992] [MRT-47937] [WARN] [dku.kernels] - Killing kernel python-single-command-kernel [2025/05/14-08:07:43.992] [MRT-47937] [INFO] [dku.containers.kubernetes] - Deleting Kubernetes object: ["delete","-f","/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/pod.yaml","-f","/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/secret.yaml"] [2025/05/14-08:07:44.092] [Thread-38200] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2025/05/14-08:07:44.092] [MRT-47937] [INFO] [dku.engine] - Schedule closing for WorkerPool [2025/05/14-08:07:44.092] [MRT-47937] [DEBUG] [dku.remoterun.registry] - Finished with container execution doctor-preprocessing-vxjysuz [2025/05/14-08:07:44.092] [AsyncCloser-773] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers] - Closing worker pool pool-6axq6lurmz74ilgg [2025/05/14-08:07:44.093] [AsyncCloser-773] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers] - Unregistered worker pool: pool-6axq6lurmz74ilgg [2025/05/14-08:07:44.093] [AsyncCloser-773] [INFO] [dku.engine] - Successfully closed WorkerPool [2025/05/14-08:07:44.093] [MRT-47937] [INFO] [dip.tickets] - Destroying API ticket for analysis-ml-TUT_TS_ANALYSIS-vxJysuz on behalf of 4131210050_rakha@pknstan.ac.id [2025/05/14-08:07:44.093] [MRT-47937] [ERROR] [dku.analysis.ml.python] - Failure while destroying ml kernel java.io.IOException: Could not stop kubernetes object: return value was 1 (Error from server (NotFound): error when deleting "/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/pod.yaml": pods "dataiku-exec-doctor-preprocessing-vxjysuz" not found Error from server (NotFound): error when deleting "/data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/secret.yaml": secrets "dkusecret-doctor-preprocessing-vxjysuz" not found ) at com.dataiku.dip.containers.exec.KubernetesExecUtils.delete(KubernetesExecUtils.java:1949) at com.dataiku.dip.containers.exec.KubernetesExecUtils.deleteFromFiles(KubernetesExecUtils.java:1934) at com.dataiku.dip.containers.exec.KubernetesExecUtils$KubernetesPod.delete(KubernetesExecUtils.java:2064) at com.dataiku.dip.containers.exec.KubernetesExecUtils$KubernetesPod.evilKill(KubernetesExecUtils.java:2079) at com.dataiku.dip.kernels.DSSKernelBase.killWithoutMercy(DSSKernelBase.java:218) at com.dataiku.dip.analysis.ml.shared.PRNSTrainThread.run(PRNSTrainThread.java:192) [2025/05/14-08:07:44.094] [MRT-47937] [INFO] [dku.block.link] - Closed socket [2025/05/14-08:07:44.094] [MRT-47937] [INFO] [dku.block.link] - Closed socket [2025/05/14-08:07:44.094] [MRT-47937] [INFO] [dku.block.link] - Closed serverSocket [2025/05/14-08:07:44.095] [MRT-47937] [ERROR] [dku.analysis.ml.python] - Processing failed com.dataiku.dip.io.SocketBlockLinkKernelException: Failed to train : : module 'numpy' has no attribute 'MachAr' at com.dataiku.dip.io.SocketBlockLinkInteraction.throwExceptionFromPython(SocketBlockLinkInteraction.java:306) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.checkException(SocketBlockLinkInteraction.java:219) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.get(SocketBlockLinkInteraction.java:194) at com.dataiku.dip.io.SingleCommandKernelLink$1.call(SingleCommandKernelLink.java:215) at com.dataiku.dip.analysis.ml.prediction.PredictionTrainAdditionalThread.process(PredictionTrainAdditionalThread.java:76) at com.dataiku.dip.analysis.ml.shared.PRNSTrainThread.run(PRNSTrainThread.java:164) [2025/05/14-08:07:44.095] [MRT-47937] [INFO] [dku.analysis.ml] - Locking model train info file /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json [2025/05/14-08:07:44.096] [MRT-47937] [INFO] [dku.analysis.ml] - Unlocked model train info file /data/dataiku/datadir/analysis-data/TUT_TS_ANALYSIS/ViMgkniW/cqxrqRei/sessions/s1/pp1/m1/train_info.json [2025/05/14-08:07:47.907] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.containers.kubernetes] - wait for pod to complete, getting pod status [2025/05/14-08:07:47.908] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-vxjysuz- try number 1 [2025/05/14-08:07:47.994] [Thread-38223] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2025/05/14-08:07:47.995] [KNL-python-single-command-kernel-k8smonitor-47962] [ERROR] [dku.containers.kubernetes] - kubectl get pod failed, no pod found : dataiku-exec-doctor-preprocessing-vxjysuz [2025/05/14-08:07:52.960] [AsyncCloser-773] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers] - Closing worker pool pool-pdsncrasxz8eezva [2025/05/14-08:07:52.961] [AsyncCloser-773] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers] - Unregistered worker pool: pool-pdsncrasxz8eezva [2025/05/14-08:07:52.961] [AsyncCloser-773] [INFO] [dku.engine] - Successfully closed WorkerPool [2025/05/14-08:07:52.964] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.ml.python] T-cqxrqRei - [ct: 18088] Processing thread joined ... [2025/05/14-08:07:52.965] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.ml.python] T-cqxrqRei - [ct: 18089] Joining processing thread ... [2025/05/14-08:07:52.995] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-vxjysuz- try number 2 [2025/05/14-08:07:53.089] [Thread-38235] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2025/05/14-08:07:53.089] [KNL-python-single-command-kernel-k8smonitor-47962] [ERROR] [dku.containers.kubernetes] - kubectl get pod failed, no pod found : dataiku-exec-doctor-preprocessing-vxjysuz [2025/05/14-08:07:58.090] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-vxjysuz- try number 3 [2025/05/14-08:07:58.176] [Thread-38241] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2025/05/14-08:07:58.176] [KNL-python-single-command-kernel-k8smonitor-47962] [ERROR] [dku.containers.kubernetes] - kubectl get pod failed, no pod found : dataiku-exec-doctor-preprocessing-vxjysuz [2025/05/14-08:08:03.177] [KNL-python-single-command-kernel-k8smonitor-47962] [WARN] [dku.containers.kubernetes] - Error getting pod status java.io.IOException: Error while checking kubernetes pod status, no pod found at com.dataiku.dip.containers.exec.KubernetesExecUtils.getPodStatus(KubernetesExecUtils.java:1735) at com.dataiku.dip.containers.exec.KubernetesExecUtils.waitForPod(KubernetesExecUtils.java:1758) at com.dataiku.dip.analysis.coreservices.AnalysisMLContainerKernel$K8SKernelMonitorThread.run(AnalysisMLContainerKernel.java:311) [2025/05/14-08:08:03.177] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.ml.kernel] - K8S pod terminated, exception:java.io.IOException: Error while checking kubernetes pod status, no pod found [2025/05/14-08:08:03.177] [KNL-python-single-command-kernel-k8smonitor-47962] [INFO] [dku.ml.kernel] - Log handling process done with code 0 [2025/05/14-08:08:03.177] [KNL-python-single-command-kernel-k8smonitor-47962] [DEBUG] [dku.resourceusage] - Reporting completion of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"4131210050_rakha@pknstan.ac.id","projectKey":"TUT_TS_ANALYSIS","analysisId":"ViMgkniW","mlTaskId":"cqxrqRei","sessionId":"s1"},"type":"SINGLE_K8S_JOB","id":"HYbf04OLO5xrkld1","startTime":1747210054963,"singleK8SJob":{"k8sClusterId":"__builtin__","executionId":"doctor-preprocessing-vxjysuz","cpuRequest":1.0,"cpuLimit":1.0,"memoryRequestMB":4096,"memoryLimitMB":4096}} [2025/05/14-08:08:14.121] [AsyncCloser-773] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers] - Closing worker pool pool-tchi6iebiqmievak [2025/05/14-08:08:14.122] [AsyncCloser-773] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers] - Unregistered worker pool: pool-tchi6iebiqmievak [2025/05/14-08:08:14.122] [AsyncCloser-773] [INFO] [dku.engine] - Successfully closed WorkerPool [2025/05/14-08:08:14.125] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.ml.python] T-cqxrqRei - [ct: 39249] Processing thread joined ... [2025/05/14-08:08:14.126] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis] T-cqxrqRei - [ct: 39250] Train done [2025/05/14-08:08:14.126] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.prediction] T-cqxrqRei - [ct: 39250] Train done [2025/05/14-08:08:14.132] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.analysis.trainingdetails] T-cqxrqRei - Publishing mltask-train-done reflected event [2025/05/14-08:08:14.135] [FT-TrainWorkThread-YgFeaaPB-47936] [INFO] [dku.futures] - [ct: 39259] Completed future execution: TrainWorkThread