[2023/12/18-16:05:05.342] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.prediction] - [ct: 0] ****************************************** [2023/12/18-16:05:05.342] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.prediction] - [ct: 0] ** Start train session s1 [2023/12/18-16:05:05.343] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.prediction] - [ct: 1] ****************************************** [2023/12/18-16:05:05.345] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.shaker.data] T-LBymz4K2 - [ct: 3] Need to compute sampleId before checking memory cache [2023/12/18-16:05:05.345] [FT-TrainWorkThread-b3mxb4vl-3759946] [DEBUG] [dip.shaker.runner] T-LBymz4K2 - [ct: 3] Script settings sampleMax=104857600 processedMax=-1 [2023/12/18-16:05:05.345] [FT-TrainWorkThread-b3mxb4vl-3759946] [DEBUG] [dip.shaker.runner] T-LBymz4K2 - [ct: 3] Processing with sampleMax=104857600 processedMax=524288000 [2023/12/18-16:05:05.346] [FT-TrainWorkThread-b3mxb4vl-3759946] [DEBUG] [dip.shaker.runner] T-LBymz4K2 - [ct: 4] Computed required sample id : 7a121235d7fc6f85e0cf3b2ddf4a640d-NA-67d2d4a7bb9f68bd5a120c6956099c590--d751713988987e9331980363e24189ce [2023/12/18-16:05:05.346] [FT-TrainWorkThread-b3mxb4vl-3759946] [DEBUG] [dku.shaker.cache] T-LBymz4K2 - Shaker MemoryCache get on dataset NETWORK_TRAFFIC_PREDICTION.seasonalcomponent key=ds=a6e54ebfd6c4d02a6ad9bcf1d20efbcd--scr=e696b1493bbeb6653664aa664c502906--samp=7a121235d7fc6f85e0cf3b2ddf4a640d-NA-67d2d4a7bb9f68bd5a120c6956099c590--d751713988987e9331980363e24189ce: hit [2023/12/18-16:05:05.346] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.shaker.schema] T-LBymz4K2 - [ct: 4] Column datum meaning=Date fail=0 [2023/12/18-16:05:05.346] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.shaker.schema] T-LBymz4K2 - [ct: 4] Column seasonal meaning=DoubleMeaning fail=0 [2023/12/18-16:05:05.348] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.datasets.bloblike] T-LBymz4K2 - [ct: 6] Enumerating blob-like dataset NETWORK_TRAFFIC_PREDICTION.seasonalcomponent prefix= [2023/12/18-16:05:05.348] [FT-TrainWorkThread-b3mxb4vl-3759946] [DEBUG] [dku.datasets.fsbased] T-LBymz4K2 - [ct: 6] Building FS provider for dataset handler: NETWORK_TRAFFIC_PREDICTION.seasonalcomponent [2023/12/18-16:05:05.348] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.fs.s3] T-LBymz4K2 - [ct: 6] Created S3 FS provider bucket=cdk-s3-bucket-iam-tdo-datafundamen-bucket83908e77-mdsyrrdmeq7d effectivePath=/dataiku/NETWORK_TRAFFIC_PREDICTION/dev/seasonalcomponent from '' and '/dataiku/NETWORK_TRAFFIC_PREDICTION/dev/seasonalcomponent' [2023/12/18-16:05:05.348] [FT-TrainWorkThread-b3mxb4vl-3759946] [DEBUG] [dku.datasets.fsbased] T-LBymz4K2 - [ct: 6] FS Provider built [2023/12/18-16:05:05.350] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.fs.s3] T-LBymz4K2 - [ct: 8] Bucket is in location eu-west-1 [2023/12/18-16:05:05.350] [FT-TrainWorkThread-b3mxb4vl-3759946] [DEBUG] [dku.fs.s3] T-LBymz4K2 - [ct: 8] Done create S3 client [2023/12/18-16:05:05.393] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.fs.s3] T-LBymz4K2 - [ct: 51] Start S3 Enumeration ON bucketPath=dataiku/NETWORK_TRAFFIC_PREDICTION/dev/seasonalcomponent/ prefix= fullPath=dataiku/NETWORK_TRAFFIC_PREDICTION/dev/seasonalcomponent/ [2023/12/18-16:05:05.488] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.fs.s3] T-LBymz4K2 - [ct: 146] S3 enumeration done, found 1 items, 7594 bytes [2023/12/18-16:05:05.489] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.input.push] T-LBymz4K2 - USTP: push selection.method=COLUMN_ORDERED records=100000 ratio=0.02 col=datum [2023/12/18-16:05:05.489] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.format] T-LBymz4K2 - [ct: 147] Extractor run: limit={"maxBytes":-1,"maxRecords":-1,"ordering":{"enabled":false,"rules":[]}} totalRecords=0 [2023/12/18-16:05:05.489] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.fs.s3] T-LBymz4K2 - [ct: 147] Getting S3 stream on dataiku/NETWORK_TRAFFIC_PREDICTION/dev/seasonalcomponent/out-s0.csv.gz [2023/12/18-16:05:05.501] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.fs.s3] T-LBymz4K2 - [ct: 159] Path to read /dataiku/NETWORK_TRAFFIC_PREDICTION/dev/seasonalcomponent|/out-s0.csv.gz -> dataiku/NETWORK_TRAFFIC_PREDICTION/dev/seasonalcomponent/out-s0.csv.gz [2023/12/18-16:05:05.501] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku] T-LBymz4K2 - getCompression filename=**out-s0.csv.gz** [2023/12/18-16:05:05.501] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.fs.s3] T-LBymz4K2 - [ct: 159] Getting range on -1 [2023/12/18-16:05:05.532] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.fs.s3] T-LBymz4K2 - [ct: 190] getBasicInputStream OK [2023/12/18-16:05:05.532] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku] T-LBymz4K2 - getCompression filename=**out-s0.csv.gz** [2023/12/18-16:05:05.532] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.format] T-LBymz4K2 - [ct: 190] Start compressed [GZIP] stream: /out-s0.csv.gz / totalRecsBefore=0 [2023/12/18-16:05:05.533] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku] T-LBymz4K2 - getCompression filename=**out-s0.csv.gz** [2023/12/18-16:05:05.533] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku] T-LBymz4K2 - getCompression filename=**out-s0.csv.gz** [2023/12/18-16:05:05.534] [FT-TrainWorkThread-b3mxb4vl-3759946] [WARN] [dip.ordered.sample] T-LBymz4K2 - Ordering column "datum" is neither numeric or date but STRING [2023/12/18-16:05:05.537] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.format] T-LBymz4K2 - [ct: 195] after stream totalComp=7594 totalUncomp=57068 totalRec=1302 [2023/12/18-16:05:05.537] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.format] T-LBymz4K2 - [ct: 195] Extractor run done, totalCompressed=7594 totalRecords=1302 [2023/12/18-16:05:05.538] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dip.sorter.chunk] T-LBymz4K2 - Spilling chunk. used=306832 [2023/12/18-16:05:05.541] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dip.ordered.sample] T-LBymz4K2 - Sorter received 1302 rows, needs to produce 100000 [2023/12/18-16:05:05.548] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dip.ordered.sample] T-LBymz4K2 - Final output got 1302 rows [2023/12/18-16:05:05.549] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.splits] T-LBymz4K2 - [ct: 207] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,streamAll=true,kfold=false,folds=5,splitBeforePrepare=true,ds=seasonalcomponent,sel=(method=column-ordered,column=datum,ascending=false,records=100000),r=0.8,s=1337, instance id: acd6f5c3cfde4c963529de155d730f95-0 [2023/12/18-16:05:05.550] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.splits] T-LBymz4K2 - [ct: 208] Search for split: p=type=SPLIT_SINGLE_DATASET,streamAll=true,kfold=false,folds=5,splitBeforePrepare=true,ds=seasonalcomponent,sel=(method=column-ordered,column=datum,ascending=false,records=100000),r=0.8,s=1337 i=acd6f5c3cfde4c963529de155d730f95-0 [2023/12/18-16:05:05.551] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.splits] T-LBymz4K2 - [ct: 209] Search for split: p=type=SPLIT_SINGLE_DATASET,streamAll=true,kfold=false,folds=5,splitBeforePrepare=true,ds=seasonalcomponent,sel=(method=column-ordered,column=datum,ascending=false,records=100000),r=0.8,s=1337 i=acd6f5c3cfde4c963529de155d730f95-0 [2023/12/18-16:05:05.551] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.splits] T-LBymz4K2 - [ct: 209] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,streamAll=true,kfold=false,folds=5,splitBeforePrepare=true,ds=seasonalcomponent,sel=(method=column-ordered,column=datum,ascending=false,records=100000),r=0.8,s=1337, instance id: acd6f5c3cfde4c963529de155d730f95-0 [2023/12/18-16:05:05.552] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.splits] T-LBymz4K2 - [ct: 210] Search for split: p=type=SPLIT_SINGLE_DATASET,streamAll=true,kfold=false,folds=5,splitBeforePrepare=true,ds=seasonalcomponent,sel=(method=column-ordered,column=datum,ascending=false,records=100000),r=0.8,s=1337 i=acd6f5c3cfde4c963529de155d730f95-0 [2023/12/18-16:05:05.553] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.splits] T-LBymz4K2 - [ct: 211] Search for split: p=type=SPLIT_SINGLE_DATASET,streamAll=true,kfold=false,folds=5,splitBeforePrepare=true,ds=seasonalcomponent,sel=(method=column-ordered,column=datum,ascending=false,records=100000),r=0.8,s=1337 i=acd6f5c3cfde4c963529de155d730f95-0 [2023/12/18-16:05:05.553] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.ml.python] T-LBymz4K2 - [ct: 211] Joining processing thread ... [2023/12/18-16:05:05.555] [MRT-3759949] [INFO] [dku.analysis.ml.python] - TrainAdditionalThread done [2023/12/18-16:05:05.554] [MRT-3759948] [INFO] [dku.analysis.ml.python] - Running a preprocessing set: pp1 in /data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1 [2023/12/18-16:05:05.556] [MRT-3759948] [INFO] [dku.block.link] - Started a socket on port 46665 [2023/12/18-16:05:05.556] [MRT-3759948] [INFO] [dku.ml.distributed.service] - New worker pool created: pool-8wxw9m7kgr6xgpzg [2023/12/18-16:05:05.557] [MRT-3759948] [INFO] [dku.python.single_command.kernel] - Attached worker pool to com.dataiku.dip.analysis.coreservices.AnalysisMLContainerKernel kernel: pool-8wxw9m7kgr6xgpzg [2023/12/18-16:05:05.557] [MRT-3759948] [INFO] [dku.ml.kernel] - Writing output of python-single-command-kernel to /data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/train.log [2023/12/18-16:05:05.557] [MRT-3759948] [DEBUG] [dku.remoterun.registry] - Registering container execution doctor-preprocessing-yjywqyj of type DOCTOR_PYTHON [2023/12/18-16:05:05.557] [MRT-3759948] [INFO] [dip.tickets] - Creating API ticket for analysis-ml-NETWORK_TRAFFIC_PREDICTION-YJYwqYJ on behalf of alexandra.verhagen@kpn.com id=analysis-ml-NETWORK_TRAFFIC_PREDICTION-YJYwqYJ_BTXsB3AWhx1A [2023/12/18-16:05:05.557] [MRT-3759948] [INFO] [dku.containerexec.imageshelper] - Finding out which image to use for requestedBase=dku-exec-base:kpn-dss-primary envLang=PYTHON envName=metadata_paid_media_39 envVersion=null [2023/12/18-16:05:05.558] [MRT-3759948] [INFO] [dku.containerexec.imageshelper] - Will use dku-exec-base-kpn-dss-primary-pyenv-metadata_paid_media_39:r-2023-11-28-12-48-01-640 [2023/12/18-16:05:05.559] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Creating Kubernetes pod from files ["/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/secret.yaml","/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/pod.yaml"] [2023/12/18-16:05:06.034] [Thread-3677291] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2023/12/18-16:05:06.034] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Starting to wait for logs to be available [2023/12/18-16:05:06.034] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Getting pod [2023/12/18-16:05:06.111] [Thread-3677298] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2023/12/18-16:05:07.111] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod is now created, waiting for logs fetch to be successful [2023/12/18-16:05:07.111] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:07.199] [Thread-3677311] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:07.200] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:07.200] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Waiting for logs, time elapsed: 89 milliseconds, status changed to: Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating [2023/12/18-16:05:07.200] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:08.200] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:08.294] [Thread-3677315] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:08.294] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:08.294] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:09.294] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:09.382] [Thread-3677360] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:09.382] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:09.382] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:10.383] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:10.479] [Thread-3677370] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:10.479] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:10.479] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:11.479] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:11.561] [Thread-3677394] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:11.561] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:11.562] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:12.562] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:12.656] [Thread-3677403] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:12.656] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:12.656] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:13.657] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:13.745] [Thread-3677409] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:13.746] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:13.746] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:14.746] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:14.841] [Thread-3677424] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:14.841] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:14.841] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:15.842] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:15.929] [Thread-3677427] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:15.930] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:15.930] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:16.930] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:17.018] [Thread-3677440] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:17.019] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:17.019] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:18.019] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:18.120] [Thread-3677443] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:18.120] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:18.121] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:19.121] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:19.219] [Thread-3677446] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:19.219] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:19.220] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:20.220] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:20.314] [Thread-3677460] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:20.314] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:20.314] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:21.315] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:21.404] [Thread-3677472] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:21.404] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:21.405] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:22.405] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:22.500] [Thread-3677493] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:22.501] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:22.501] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:23.501] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:23.595] [Thread-3677499] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:23.596] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:23.596] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:24.596] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:24.690] [Thread-3677508] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:24.690] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:24.691] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:25.691] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:25.782] [Thread-3677517] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:25.783] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:25.783] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:26.783] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:26.878] [Thread-3677526] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:26.879] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:26.879] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:27.879] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:27.980] [Thread-3677529] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:27.981] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:27.981] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:28.981] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:29.079] [Thread-3677532] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:29.079] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:29.079] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:30.079] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:30.240] [Thread-3677544] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:30.241] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:30.241] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:31.241] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:31.335] [Thread-3677553] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:31.335] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:31.335] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:32.336] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:32.425] [Thread-3677577] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:32.425] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:32.426] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:33.426] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:33.518] [Thread-3677583] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:33.519] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:33.519] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:34.519] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:34.606] [Thread-3677592] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:34.607] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:34.607] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:35.607] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:35.704] [Thread-3677601] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:35.704] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:35.704] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:36.705] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:36.795] [Thread-3677610] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:36.795] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:36.795] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:37.796] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:37.899] [Thread-3677613] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:37.899] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:37.899] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:38.900] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:38.996] [Thread-3677616] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:38.996] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:38.996] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:39.996] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:40.103] [Thread-3677625] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:40.103] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:40.103] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:41.104] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:41.202] [Thread-3677637] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:41.202] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:41.202] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:42.203] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:42.293] [Thread-3677655] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:42.293] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:42.293] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:43.294] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:43.383] [Thread-3677667] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:43.384] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:43.384] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:44.384] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:44.473] [Thread-3677673] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:44.473] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:44.473] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:45.473] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:45.564] [Thread-3677685] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:45.564] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:45.564] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:46.565] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:46.670] [Thread-3677692] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:46.670] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:46.671] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:47.671] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:47.769] [Thread-3677698] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:47.770] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:47.770] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:48.770] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:48.866] [Thread-3677701] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:48.866] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:48.867] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:49.867] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:49.959] [Thread-3677707] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:49.959] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:49.959] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:50.959] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:51.046] [Thread-3677722] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:51.046] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:51.047] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:52.047] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:52.138] [Thread-3677740] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:05:52.138] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Get logs failed [2023/12/18-16:05:52.139] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Pod has not started yet: 'Error from server (BadRequest): container "c" in pod "dataiku-exec-doctor-preprocessing-yjywqyj" is waiting to start: ContainerCreating ' [2023/12/18-16:05:53.139] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Calling logs on pods/dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:05:53.259] [Thread-3677749] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2023/12/18-16:05:53.259] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Getting logs succeeded, but checking that pod is indeed running [2023/12/18-16:05:53.372] [Thread-3677752] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2023/12/18-16:05:53.373] [MRT-3759948] [INFO] [dku.security.process] - Starting process (regular) [2023/12/18-16:05:53.375] [MRT-3759948] [INFO] [dku.security.process] - Process started with pid=4068240 [2023/12/18-16:05:53.376] [MRT-3759948] [DEBUG] [dku.resourceusage] - Reporting start of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"alexandra.verhagen@kpn.com","projectKey":"NETWORK_TRAFFIC_PREDICTION","analysisId":"HjNevZBL","mlTaskId":"LBymz4K2","sessionId":"s1"},"type":"SINGLE_K8S_JOB","id":"CJlq4xxHFjZ43a9T","startTime":1702915505557,"singleK8SJob":{"k8sClusterId":"__builtin__","executionId":"doctor-preprocessing-yjywqyj","cpuRequest":-1.0,"cpuLimit":-1.0,"memoryRequestMB":-1,"memoryLimitMB":-1}} [2023/12/18-16:05:53.377] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.ml.kernel] - Waiting for K8S pod termination [2023/12/18-16:05:53.378] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - wait for pod to complete, getting pod status [2023/12/18-16:05:53.378] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-yjywqyj- try number 1 [2023/12/18-16:05:53.479] [Thread-3677761] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2023/12/18-16:05:53.479] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - kubectl get pod OK, out=Running [2023/12/18-16:05:53.479] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - Kubernetes pod is running [2023-12-18 16:05:53,564] [1/MainThread] [INFO] [root] Fetching job definition [2023-12-18 16:05:53,564] [1/MainThread] [DEBUG] [root] Creating HTTP session for internal RPC [2023-12-18 16:05:53,564] [1/MainThread] [DEBUG] [root] Verifying SSL calls: /home/dataiku/rpc_server_cert.pem [2023-12-18 16:05:53,566] [1/MainThread] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): ip-10-12-45-251.eu-west-1.compute.internal:10001 [2023-12-18 16:05:53,576] [1/MainThread] [DEBUG] [urllib3.connectionpool] http://ip-10-12-45-251.eu-west-1.compute.internal:10001 "POST /dip/api/tintercom/containers/get-execution HTTP/1.1" 200 None /opt/dataiku/python/dataiku/container/runner.py:799: DeprecationWarning: The 'warn' function is deprecated, use 'warning' instead logging.warn("Container image was build with version %s, but execution was sent from DSS version %s" [2023-12-18 16:05:53,577] [1/MainThread] [WARNING] [root] Container image was build with version 12.1.3, but execution was sent from DSS version 12.3.2 [2023-12-18 16:05:53,578] [1/MainThread] [INFO] [root] got exec: {"dssVersion": "12.3.2", "id": "doctor-preprocessing-yjywqyj", "authCtx": {"userGroupLevelPermissions": {"admin": false, "mayManageUDM": false, "mayCreateProjects": false, "mayCreateProjectsFromMacros": false, "mayCreateProjectsFromTemplates": false, "mayCreateProjectsFromDataikuApps": false, "mayWriteUnsafeCode": false, "mayWriteSafeCode": true, "mayCreateAuthenticatedConnections": false, "mayCreateCodeEnvs": false, "mayCreateClusters": false, "mayCreateCodeStudioTemplates": false, "mayDevelopPlugins": false, "mayEditLibFolders": false, "mayManageCodeEnvs": false, "mayManageClusters": false, "mayManageCodeStudioTemplates": false, "mayViewIndexedHiveConnections": false, "mayCreatePublishedAPIServices": true, "mayCreatePublishedProjects": true, "mayWriteInRootProjectFolder": true, "mayCreateActiveWebContent": true, "mayCreateWorkspaces": false, "mayShareToWorkspaces": true, "mayCreateDataCollections": true, "mayPublishToDataCollections": true, "mayManageFeatureStore": false}, "via": [], "authSource": "USER_FROM_UI", "realUserLogin": "alexandra.verhagen@kpn.com", "realUserEmail": "alexandra.verhagen@kpn.com", "userGroups": ["DATASCIENTIST_TDO"], "userProfile": "DESIGNER"}, "projectKey": "NETWORK_TRAFFIC_PREDICTION", "backendDipHomePath": "/data/dataiku/dss_data", "contextPath": "/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2", "executionPath": "/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1", "type": "DOCTOR_PYTHON", "definition": null, "payload": null, "codeEnvs": [{"lang": "PYTHON", "name": "metadata_paid_media_39", "version": "Latest"}], "envResource": null} [2023-12-18 16:05:53,578] [1/MainThread] [INFO] [root] Training doctor model [2023-12-18 16:05:53,578] [1/MainThread] [INFO] [root] Fetching project libraries [2023-12-18 16:05:53,593] [1/MainThread] [DEBUG] [urllib3.connectionpool] http://ip-10-12-45-251.eu-west-1.compute.internal:10001 "POST /dip/api/tintercom/containers/get-file HTTP/1.1" 200 None [2023-12-18 16:05:53,599] [1/MainThread] [INFO] [root] Fetching instance libraries [2023-12-18 16:05:53,603] [1/MainThread] [DEBUG] [urllib3.connectionpool] http://ip-10-12-45-251.eu-west-1.compute.internal:10001 "POST /dip/api/tintercom/containers/get-file HTTP/1.1" 200 None [2023-12-18 16:05:53,606] [1/MainThread] [DEBUG] [root] Adding /home/dataiku/lib/project/project-python-libs/NETWORK_TRAFFIC_PREDICTION/python to Pythonpath [2023-12-18 16:05:53,606] [1/MainThread] [DEBUG] [root] Adding /home/dataiku/lib/project/project-python-libs/NETWORK_TRAFFIC_PREDICTION/dss-general-functionality to Pythonpath [2023-12-18 16:05:53,607] [1/MainThread] [DEBUG] [root] Adding /home/dataiku/lib/project/project-python-libs/NETWORK_TRAFFIC_PREDICTION/dataloader to Pythonpath [2023-12-18 16:05:53,607] [1/MainThread] [INFO] [root] running on python 3.9.17 (main, Oct 31 2023, 12:01:31) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] [2023-12-18 16:05:53,607] [1/MainThread] [INFO] [root] now using PYTHONPATH: /home/dataiku:/opt/dataiku/python:/home/dataiku/lib/instance:/home/dataiku/lib/project/project-python-libs/NETWORK_TRAFFIC_PREDICTION/python:/home/dataiku/lib/project/project-python-libs/NETWORK_TRAFFIC_PREDICTION/dss-general-functionality:/home/dataiku/lib/project/project-python-libs/NETWORK_TRAFFIC_PREDICTION/dataloader [2023-12-18 16:05:53,607] [1/MainThread] [INFO] [root] Fetching dir into work [2023-12-18 16:05:53,610] [1/MainThread] [DEBUG] [urllib3.connectionpool] http://ip-10-12-45-251.eu-west-1.compute.internal:10001 "POST /dip/api/tintercom/containers/get-file HTTP/1.1" 200 None [2023-12-18 16:05:53,611] [1/MainThread] [INFO] [root] Running doctor server [2023-12-18 16:05:53,617] [1/MainThread] [INFO] [root] Running subprocess in PID 45 [2023-12-18 16:05:53,617] [1/Thread-1] [INFO] [root] Start monitoring process 45 [2023/12/18-16:05:55.156] [MRT-3759948] [INFO] [dku.link.secret_protected] - Connected to kernel [2023/12/18-16:05:55.157] [MRT-3759948] [INFO] [dku.block.link.interaction] - Execute link command respClazz=true respTypeToken=false respIsString=false is=false asyncInputStream=false os=false /opt/dataiku/python/dataiku/container/runner.py:479: DeprecationWarning: invalid escape sequence \s """ % (access_token_cookie_name, access_token_cookie_name) /opt/dataiku/python/dataiku/container/runner.py:554: DeprecationWarning: invalid escape sequence \: """ % (access_token_purge, inside_port) /opt/dataiku/python/dataiku/container/runner.py:700: DeprecationWarning: invalid escape sequence \: """ % (inside_port) [2023-12-18 16:05:55,153] [45/MainThread] [INFO] [root] Launching doctor command server [2023-12-18 16:05:55,155] [45/MainThread] [INFO] [dataiku.base.socket_block_link] Connecting to ip-10-12-45-251.eu-west-1.compute.internal (10.12.45.251) at port 46665 [2023-12-18 16:05:55,156] [45/MainThread] [INFO] [dataiku.base.socket_block_link] Connected to ip-10-12-45-251.eu-west-1.compute.internal (10.12.45.251) at port 46665 [2023-12-18 16:05:55,420] [45/MainThread] [INFO] [root] Running analysis command: train_prediction_timeseries [2023/12/18-16:05:55.442] [MRT-3759948] [INFO] [dku.block.link.interaction] - Check result for nullity exceptionIfNull=true result=null /opt/dataiku/python/dataiku/doctor/timeseries/train/split_handler.py:109: DeprecationWarning: invalid escape sequence \ """ /opt/dataiku/python/dataiku/doctor/timeseries/train/split_handler.py:146: DeprecationWarning: invalid escape sequence \ """ [2023-12-18 16:05:55,442] [45/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: DatasetSanityCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_DATASET_SANITY_CHECKS [2023-12-18 16:05:55,442] [45/MainThread] [INFO] [dataiku.doctor.utils.listener] START - Loading source dataset Traceback (most recent call last): File "/opt/dataiku/python/dataiku/doctor/server.py", line 45, in serve ret = api_command(arg) File "/opt/dataiku/python/dataiku/doctor/dkuapi.py", line 45, in aux return api(**kwargs) File "/opt/dataiku/python/dataiku/doctor/commands.py", line 782, in train_prediction_timeseries launch_training(core_params, modeling_sets, preprocessing_params, resampling_params, run_folder, split_desc) File "/opt/dataiku/python/dataiku/doctor/timeseries/train/launch_training.py", line 59, in launch_training with preprocessing_listener.push_step(step_constants.ProcessingStep.STEP_LOADING_SRC): File "/opt/dataiku/python/dataiku/doctor/utils/listener.py", line 37, in __enter__ self.listener.save_status() File "/opt/dataiku/python/dataiku/doctor/utils/listener.py", line 130, in save_status child.save_status() File "/opt/dataiku/python/dataiku/doctor/utils/listener.py", line 128, in save_status self.context.save_status(self, self.parent) File "/opt/dataiku/python/dataiku/doctor/utils/listener.py", line 84, in save_status utils.write_running_traininfo(self.folder, self.start, listeners) File "/opt/dataiku/python/dataiku/doctor/utils/__init__.py", line 297, in write_running_traininfo dkujson.dump_to_filepath(status_filepath, status) File "/opt/dataiku/python/dataiku/core/dkujson.py", line 56, in dump_to_filepath with open(filepath, 'w') as f: FileNotFoundError: [Errno 2] No such file or directory: '/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/m1/train_info.json' [2023-12-18 16:05:55,443] [45/MainThread] [INFO] [dataiku.base.socket_block_link] Client closed [2023/12/18-16:05:55.943] [MRT-3759948] [INFO] [dku.kernels] - Getting kernel tail [2023/12/18-16:05:55.946] [MRT-3759948] [INFO] [dku.ml.kernel] - Kubernetes: Trying to enrich exception: com.dataiku.dip.io.SocketBlockLinkKernelException: Failed to train : : [Errno 2] No such file or directory: '/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/m1/train_info.json' from kernel com.dataiku.dip.analysis.coreservices.AnalysisMLContainerKernel@432ba877 process=com.dataiku.dip.containers.exec.KubernetesExecUtils$KubernetesPod@5d554a9f pid=4068240 retcode=null [2023/12/18-16:05:55.946] [MRT-3759948] [WARN] [dku.analysis.ml.python] - Training failed com.dataiku.dip.io.SocketBlockLinkKernelException: Failed to train : : [Errno 2] No such file or directory: '/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/m1/train_info.json' at com.dataiku.dip.io.SocketBlockLinkInteraction.throwExceptionFromPython(SocketBlockLinkInteraction.java:302) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.checkException(SocketBlockLinkInteraction.java:215) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.get(SocketBlockLinkInteraction.java:190) at com.dataiku.dip.io.SingleCommandKernelLink$1.call(SingleCommandKernelLink.java:211) at com.dataiku.dip.analysis.ml.prediction.PredictionTrainAdditionalThread.process(PredictionTrainAdditionalThread.java:76) at com.dataiku.dip.analysis.ml.shared.PRNSTrainThread.run(PRNSTrainThread.java:167) [2023/12/18-16:05:55.947] [MRT-3759948] [INFO] [dku.ml.kernel] - Waiting for error after pod failed [2023/12/18-16:05:55.947] [MRT-3759948] [INFO] [dku.ml.kernel] - /data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/error.json not here, still waiting [2023-12-18 16:05:57,155] [45/Thread-1] [INFO] [root] Sending 6 globs to train_infos.tgz [2023-12-18 16:05:57,155] [45/Thread-1] [INFO] [root] Sending files to train_infos.tgz: ['m1/train_info.json'] [2023-12-18 16:05:57,156] [45/Thread-1] [DEBUG] [root] Creating HTTP session for internal RPC [2023-12-18 16:05:57,157] [45/Thread-1] [DEBUG] [root] Verifying SSL calls: /home/dataiku/work/rpc_server_cert.pem [2023-12-18 16:05:57,161] [45/Thread-1] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): ip-10-12-45-251.eu-west-1.compute.internal:10001 [2023-12-18 16:05:57,169] [45/Thread-1] [DEBUG] [urllib3.connectionpool] http://ip-10-12-45-251.eu-west-1.compute.internal:10001 "POST /dip/api/tintercom/containers/put-file?executionId=doctor-preprocessing-yjywqyj&fileKind=EXECUTION_DIR&path=train_infos.tgz&expand=true HTTP/1.1" 200 2 [2023-12-18 16:05:57,170] [45/MainThread] [INFO] [root] Sending 6 globs to train_infos.tgz [2023-12-18 16:05:57,170] [45/MainThread] [INFO] [root] Sending files to train_infos.tgz: ['rpreprocessing_params.json', 'm1', 'secret.yaml', 'pod.yaml', 'remote-run-env-def.json', 'rpc_server_cert.pem'] [2023-12-18 16:05:57,181] [45/MainThread] [DEBUG] [urllib3.connectionpool] http://ip-10-12-45-251.eu-west-1.compute.internal:10001 "POST /dip/api/tintercom/containers/put-file?executionId=doctor-preprocessing-yjywqyj&fileKind=EXECUTION_DIR&path=train_infos.tgz&expand=true HTTP/1.1" 200 2 Installing debugging signal handler [2023-12-18 16:05:57,409] [1/MainThread] [INFO] [root] Stop monitoring process 45 [2023/12/18-16:05:58.447] [MRT-3759948] [INFO] [dku.ml.kernel] - /data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/error.json not here, still waiting [2023/12/18-16:05:58.479] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - wait for pod to complete, getting pod status [2023/12/18-16:05:58.480] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-yjywqyj- try number 1 [2023/12/18-16:05:58.559] [Thread-3677790] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2023/12/18-16:05:58.560] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - kubectl get pod OK, out=Running [2023/12/18-16:05:58.560] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - Kubernetes pod is running [2023-12-18 16:05:58,623] [1/Thread-1] [WARNING] [root] Process not yet started, or shutting down [2023-12-18 16:05:58,623] [1/Thread-1] [INFO] [root] Done monitoring process 45 [2023-12-18 16:05:58,623] [1/MainThread] [INFO] [root] Containerized process exited with status 0 [2023-12-18 16:05:58,629] [1/MainThread] [DEBUG] [urllib3.connectionpool] http://ip-10-12-45-251.eu-west-1.compute.internal:10001 "POST /dip/api/tintercom/containers/put-file?executionId=doctor-preprocessing-yjywqyj&fileKind=EXECUTION_DIR&path=container_done.txt HTTP/1.1" 200 2 Installing debugging signal handler [2023/12/18-16:06:00.948] [MRT-3759948] [INFO] [dku.ml.kernel] - Didn't get error file [2023/12/18-16:06:00.948] [MRT-3759948] [INFO] [dku.ml.kernel] - Deleting pod [2023/12/18-16:06:00.948] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Deleting Kubernetes object: ["delete","-f","/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/secret.yaml","-f","/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/pod.yaml"] [2023/12/18-16:06:03.560] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - wait for pod to complete, getting pod status [2023/12/18-16:06:03.560] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-yjywqyj- try number 1 [2023/12/18-16:06:03.626] [Thread-3677836] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2023/12/18-16:06:03.627] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - kubectl get pod OK, out=Running [2023/12/18-16:06:03.627] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - Kubernetes pod is running [2023/12/18-16:06:08.627] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - wait for pod to complete, getting pod status [2023/12/18-16:06:08.627] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-yjywqyj- try number 1 [2023/12/18-16:06:08.693] [Thread-3677905] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2023/12/18-16:06:08.694] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - kubectl get pod OK, out=Running [2023/12/18-16:06:08.694] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - Kubernetes pod is running [2023/12/18-16:06:13.694] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - wait for pod to complete, getting pod status [2023/12/18-16:06:13.694] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-yjywqyj- try number 1 [2023/12/18-16:06:13.760] [Thread-3677948] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2023/12/18-16:06:13.760] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - kubectl get pod OK, out=Running [2023/12/18-16:06:13.760] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.containers.kubernetes] - Kubernetes pod is running [2023/12/18-16:06:17.259] [Thread-3677805] [DEBUG] [dku.utils] - Process kubectl done (return code 0) [2023/12/18-16:06:17.260] [MRT-3759948] [INFO] [dku.ml.kernel] - Pod deleted [2023/12/18-16:06:17.260] [MRT-3759948] [WARN] [dku.kernels] - Killing kernel python-single-command-kernel [2023/12/18-16:06:17.260] [MRT-3759948] [INFO] [dku.containers.kubernetes] - Deleting Kubernetes object: ["delete","-f","/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/secret.yaml","-f","/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/pod.yaml"] [2023/12/18-16:06:17.351] [Thread-3677969] [INFO] [dku.utils] - Process kubectl completed, return code 1 [2023/12/18-16:06:17.352] [MRT-3759948] [INFO] [dku.engine] - Schedule closing for RemoteWorkerPool [2023/12/18-16:06:17.352] [MRT-3759948] [DEBUG] [dku.remoterun.registry] - Finished with container execution doctor-preprocessing-yjywqyj [2023/12/18-16:06:17.352] [AsyncCloser-291] [INFO] [dku.ml.distributed.pool] - Closing worker pool pool-8wxw9m7kgr6xgpzg [2023/12/18-16:06:17.353] [AsyncCloser-291] [INFO] [dku.ml.distributed.service] - Unregistered worker pool: pool-8wxw9m7kgr6xgpzg [2023/12/18-16:06:17.352] [MRT-3759948] [INFO] [dip.tickets] - Destroying API ticket for analysis-ml-NETWORK_TRAFFIC_PREDICTION-YJYwqYJ on behalf of alexandra.verhagen@kpn.com [2023/12/18-16:06:17.353] [AsyncCloser-291] [INFO] [dku.engine] - Successfully closed RemoteWorkerPool [2023/12/18-16:06:17.353] [MRT-3759948] [ERROR] [dku.analysis.ml.python] - Failure while destroying ml kernel java.io.IOException: Could not stop kubernetes object: return value was 1 (Error from server (NotFound): error when deleting "/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/secret.yaml": secrets "dkusecret-doctor-preprocessing-yjywqyj" not found Error from server (NotFound): error when deleting "/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/pod.yaml": pods "dataiku-exec-doctor-preprocessing-yjywqyj" not found ) at com.dataiku.dip.containers.exec.KubernetesExecUtils.delete(KubernetesExecUtils.java:1696) at com.dataiku.dip.containers.exec.KubernetesExecUtils$KubernetesPod.delete(KubernetesExecUtils.java:1829) at com.dataiku.dip.containers.exec.KubernetesExecUtils$KubernetesPod.evilKill(KubernetesExecUtils.java:1843) at com.dataiku.dip.kernels.DSSKernelBase.killWithoutMercy(DSSKernelBase.java:216) at com.dataiku.dip.analysis.ml.shared.PRNSTrainThread.run(PRNSTrainThread.java:195) [2023/12/18-16:06:17.353] [MRT-3759948] [INFO] [dku.block.link] - Closed socket [2023/12/18-16:06:17.353] [MRT-3759948] [INFO] [dku.block.link] - Closed socket [2023/12/18-16:06:17.354] [MRT-3759948] [INFO] [dku.block.link] - Closed serverSocket [2023/12/18-16:06:17.354] [MRT-3759948] [ERROR] [dku.analysis.ml.python] - Processing failed com.dataiku.dip.io.SocketBlockLinkKernelException: Failed to train : : [Errno 2] No such file or directory: '/data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/m1/train_info.json' at com.dataiku.dip.io.SocketBlockLinkInteraction.throwExceptionFromPython(SocketBlockLinkInteraction.java:302) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.checkException(SocketBlockLinkInteraction.java:215) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.get(SocketBlockLinkInteraction.java:190) at com.dataiku.dip.io.SingleCommandKernelLink$1.call(SingleCommandKernelLink.java:211) at com.dataiku.dip.analysis.ml.prediction.PredictionTrainAdditionalThread.process(PredictionTrainAdditionalThread.java:76) at com.dataiku.dip.analysis.ml.shared.PRNSTrainThread.run(PRNSTrainThread.java:167) [2023/12/18-16:06:17.354] [MRT-3759948] [INFO] [dku.analysis.ml] - Locking model train info file /data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/m1/train_info.json [2023/12/18-16:06:17.355] [MRT-3759948] [INFO] [dku.analysis.ml] - Unlocked model train info file /data/dataiku/dss_data/analysis-data/NETWORK_TRAFFIC_PREDICTION/HjNevZBL/LBymz4K2/sessions/s1/pp1/m1/train_info.json [2023/12/18-16:06:17.355] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.ml.python] T-LBymz4K2 - [ct: 72013] Processing thread joined ... [2023/12/18-16:06:17.356] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.ml.python] T-LBymz4K2 - [ct: 72014] Joining processing thread ... [2023/12/18-16:06:17.356] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.ml.python] T-LBymz4K2 - [ct: 72014] Processing thread joined ... [2023/12/18-16:06:17.356] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis] T-LBymz4K2 - [ct: 72014] Train done [2023/12/18-16:06:17.356] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.prediction] T-LBymz4K2 - [ct: 72014] Train done [2023/12/18-16:06:17.358] [FT-TrainWorkThread-b3mxb4vl-3759946] [INFO] [dku.analysis.trainingdetails] T-LBymz4K2 - Publishing mltask-train-done reflected event [2023/12/18-16:06:18.760] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - wait for pod to complete, getting pod status [2023/12/18-16:06:18.761] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-yjywqyj- try number 1 [2023/12/18-16:06:18.843] [Thread-3677972] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2023/12/18-16:06:18.843] [KNL-python-single-command-kernel-k8smonitor-3760420] [ERROR] [dku.containers.kubernetes] - kubectl get pod failed, no pod found : dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:06:23.844] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-yjywqyj- try number 2 [2023/12/18-16:06:23.950] [Thread-3678014] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2023/12/18-16:06:23.950] [KNL-python-single-command-kernel-k8smonitor-3760420] [ERROR] [dku.containers.kubernetes] - kubectl get pod failed, no pod found : dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:06:28.951] [KNL-python-single-command-kernel-k8smonitor-3760420] [INFO] [dku.containers.kubernetes] - Querying status on pod dataiku-exec-doctor-preprocessing-yjywqyj- try number 3 [2023/12/18-16:06:29.032] [Thread-3678035] [DEBUG] [dku.utils] - Process kubectl-get-pod done (return code 0) [2023/12/18-16:06:29.032] [KNL-python-single-command-kernel-k8smonitor-3760420] [ERROR] [dku.containers.kubernetes] - kubectl get pod failed, no pod found : dataiku-exec-doctor-preprocessing-yjywqyj [2023/12/18-16:06:34.033] [KNL-python-single-command-kernel-k8smonitor-3760420] [ERROR] [dku.ml.kernel] - Failure while monitoring K8S pod java.io.IOException: Error while checking kubernetes pod status, no pod found at com.dataiku.dip.containers.exec.KubernetesExecUtils.getPodStatus(KubernetesExecUtils.java:1509) at com.dataiku.dip.containers.exec.KubernetesExecUtils.waitForPod(KubernetesExecUtils.java:1530) at com.dataiku.dip.analysis.coreservices.AnalysisMLContainerKernel$K8SKernelMonitorThread.run(AnalysisMLContainerKernel.java:324) [2023/12/18-16:06:34.033] [KNL-python-single-command-kernel-k8smonitor-3760420] [DEBUG] [dku.resourceusage] - Reporting completion of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"alexandra.verhagen@kpn.com","projectKey":"NETWORK_TRAFFIC_PREDICTION","analysisId":"HjNevZBL","mlTaskId":"LBymz4K2","sessionId":"s1"},"type":"SINGLE_K8S_JOB","id":"CJlq4xxHFjZ43a9T","startTime":1702915505557,"singleK8SJob":{"k8sClusterId":"__builtin__","executionId":"doctor-preprocessing-yjywqyj","cpuRequest":-1.0,"cpuLimit":-1.0,"memoryRequestMB":-1,"memoryLimitMB":-1}}