[20240219-011814.643] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.prediction] - [ct 0] [20240219-011814.644] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.prediction] - [ct 1] Start train session s1 [20240219-011814.644] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.prediction] - [ct 1] [20240219-011814.653] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.shaker.data] T-2kk5qXQM - [ct 10] Need to compute sampleId before checking memory cache [20240219-011814.653] [FT-TrainWorkThread-GjmjUROB-93700] [DEBUG] [dip.shaker.runner] T-2kk5qXQM - [ct 10] Script settings sampleMax=104857600 processedMax=-1 [20240219-011814.653] [FT-TrainWorkThread-GjmjUROB-93700] [DEBUG] [dip.shaker.runner] T-2kk5qXQM - [ct 10] Processing with sampleMax=104857600 processedMax=4194304000 [20240219-011814.653] [FT-TrainWorkThread-GjmjUROB-93700] [DEBUG] [dip.shaker.runner] T-2kk5qXQM - [ct 10] Computed required sample id 8c26f0c57e5ea52d9873835329439dbc-NA-ac2e0aa81c1215a34ba9f85052ba5ff71708305383589--d751713988987e9331980363e24189ce [20240219-011814.654] [FT-TrainWorkThread-GjmjUROB-93700] [DEBUG] [dku.shaker.cache] T-2kk5qXQM - Shaker MemoryCache get on dataset LIU_PRACTICE.data_cleaned___kopi_prepared key=ds=626924856e139dee53bcf9ba887b475b--scr=4853e37a52fd9866ff8ef159042dd63c--samp=8c26f0c57e5ea52d9873835329439dbc-NA-ac2e0aa81c1215a34ba9f85052ba5ff71708305383589--d751713988987e9331980363e24189ce hit [20240219-011814.654] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.shaker.schema] T-2kk5qXQM - [ct 11] Column DateTime meaning=Date fail=0 [20240219-011814.654] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.shaker.schema] T-2kk5qXQM - [ct 11] Column Q meaning=LongMeaning fail=0 [20240219-011814.654] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.shaker.schema] T-2kk5qXQM - [ct 11] Column Q1 meaning=LongMeaning fail=0 [20240219-011814.656] [Thread-45294] [INFO] [dku.datasets.pull] - pull background thread starting for data_cleaned___kopi_prepared [20240219-011814.657] [Thread-45294] [INFO] [dku.datasets.bloblike] - Enumerating blob-like dataset LIU_PRACTICE.data_cleaned___kopi_prepared prefix= [20240219-011814.657] [Thread-45294] [DEBUG] [dku.datasets.fsbased] - Building FS provider for dataset handler LIU_PRACTICE.data_cleaned___kopi_prepared [20240219-011814.657] [Thread-45294] [DEBUG] [dku.fs.s3] - Created S3 provider bucket=dataiku-ts pathInBucket=dataikuLIU_PRACTICEdata_cleaned___kopi_prepared (connectionChroot=null) [20240219-011814.657] [Thread-45294] [DEBUG] [dku.datasets.fsbased] - FS Provider built [20240219-011814.658] [Thread-45294] [INFO] [dku.fs.s3] - Bucket is in location ap-northeast-1 [20240219-011814.659] [Thread-45294] [DEBUG] [dku.fs.s3] - Done create S3 client [20240219-011814.698] [Thread-45294] [INFO] [dku.fs.s3] - Start S3 Enumeration ON bucketPath=dataikuLIU_PRACTICEdata_cleaned___kopi_prepared prefix= fullPath=dataikuLIU_PRACTICEdata_cleaned___kopi_prepared [20240219-011814.894] [Thread-45294] [INFO] [dku.fs.s3] - S3 enumeration done, found 1 items, 67280 bytes [20240219-011814.894] [Thread-45294] [INFO] [dku.input.push] - USTP push selection.method=HEAD_SEQUENTIAL records=100000 ratio=0.02 col=null [20240219-011814.894] [Thread-45294] [INFO] [dku.format] - Extractor run limit={maxBytes-1,maxRecords100000,ordering{enabledfalse,rules[]}} totalRecords=0 [20240219-011814.895] [Thread-45294] [INFO] [dku.fs.s3] - Getting S3 stream on dataikuLIU_PRACTICEdata_cleaned___kopi_preparedout-s0.csv.gz [20240219-011814.907] [Thread-45294] [INFO] [dku.fs.s3] - Path to read dataikuLIU_PRACTICEdata_cleaned___kopi_preparedout-s0.csv.gz - dataikuLIU_PRACTICEdata_cleaned___kopi_preparedout-s0.csv.gz [20240219-011814.907] [Thread-45294] [INFO] [dku] - getCompression filename=out-s0.csv.gz [20240219-011814.907] [Thread-45294] [INFO] [dku.fs.s3] - Getting range on -1 [20240219-011814.936] [Thread-45294] [INFO] [dku.fs.s3] - getBasicInputStream OK [20240219-011814.936] [Thread-45294] [INFO] [dku] - getCompression filename=out-s0.csv.gz [20240219-011814.936] [Thread-45294] [INFO] [dku.format] - Start compressed [GZIP] stream out-s0.csv.gz totalRecsBefore=0 [20240219-011814.936] [Thread-45294] [INFO] [dku] - getCompression filename=out-s0.csv.gz [20240219-011814.936] [Thread-45294] [INFO] [dku] - getCompression filename=out-s0.csv.gz [20240219-011814.974] [Thread-45294] [INFO] [dku.format] - after stream totalComp=67280 totalUncomp=245472 totalRec=7077 [20240219-011814.974] [Thread-45294] [INFO] [dku.format] - Extractor run done, totalCompressed=67280 totalRecords=7077 [20240219-011814.974] [Thread-45294] [DEBUG] [dku.datasets.pull] - pull background thread ending queue, cursize=0 [20240219-011814.974] [Thread-45294] [INFO] [dku.datasets.pull] - pull background thread finished for data_cleaned___kopi_prepared [20240219-011814.974] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.datasets.pull] T-2kk5qXQM - End of stream reached [20240219-011814.975] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dip.sorter.chunk] T-2kk5qXQM - Spilling chunk. used=1835574 [20240219-011815.030] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.ml.prediction.split] T-2kk5qXQM - [ct 387] Sorted traintest split threshold = 6198 [20240219-011815.033] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.splits] T-2kk5qXQM - [ct 390] Checking if splits are up to date. Policy type=SPLIT_SINGLE_DATASET,split=SORTED,splitBeforePrepare=true,ds=data_cleaned___kopi_prepared,sel=(method=head-s,records=100000),streamAll=true,c=Q,ascending=true, instance id c50d189687948c3fee056cac0d3e81c0-0 [20240219-011815.033] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.splits] T-2kk5qXQM - [ct 390] Search for split p=type=SPLIT_SINGLE_DATASET,split=SORTED,splitBeforePrepare=true,ds=data_cleaned___kopi_prepared,sel=(method=head-s,records=100000),streamAll=true,c=Q,ascending=true i=c50d189687948c3fee056cac0d3e81c0-0 [20240219-011815.033] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.splits] T-2kk5qXQM - [ct 390] Search for split p=type=SPLIT_SINGLE_DATASET,split=SORTED,splitBeforePrepare=true,ds=data_cleaned___kopi_prepared,sel=(method=head-s,records=100000),streamAll=true,c=Q,ascending=true i=c50d189687948c3fee056cac0d3e81c0-0 [20240219-011815.034] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.splits] T-2kk5qXQM - [ct 391] Checking if splits are up to date. Policy type=SPLIT_SINGLE_DATASET,split=SORTED,splitBeforePrepare=true,ds=data_cleaned___kopi_prepared,sel=(method=head-s,records=100000),streamAll=true,c=Q,ascending=true, instance id c50d189687948c3fee056cac0d3e81c0-0 [20240219-011815.034] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.splits] T-2kk5qXQM - [ct 391] Search for split p=type=SPLIT_SINGLE_DATASET,split=SORTED,splitBeforePrepare=true,ds=data_cleaned___kopi_prepared,sel=(method=head-s,records=100000),streamAll=true,c=Q,ascending=true i=c50d189687948c3fee056cac0d3e81c0-0 [20240219-011815.035] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.splits] T-2kk5qXQM - [ct 392] Search for split p=type=SPLIT_SINGLE_DATASET,split=SORTED,splitBeforePrepare=true,ds=data_cleaned___kopi_prepared,sel=(method=head-s,records=100000),streamAll=true,c=Q,ascending=true i=c50d189687948c3fee056cac0d3e81c0-0 [20240219-011815.035] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.ml.python] T-2kk5qXQM - [ct 392] Joining processing thread ... [20240219-011815.036] [MRT-93703] [INFO] [dku.analysis.ml.python] - TrainAdditionalThread done [20240219-011815.036] [MRT-93702] [INFO] [dku.analysis.ml.python] - Running a preprocessing set pp1 in datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1 [20240219-011815.036] [MRT-93702] [INFO] [dku.block.link] - Started a socket on port 40459 [20240219-011815.036] [MRT-93702] [INFO] [dku.ml.distributed.service] - New worker pool created pool-zyxqmrmma2jtjakn [20240219-011815.037] [MRT-93702] [INFO] [dku.python.single_command.kernel] - Attached worker pool to com.dataiku.dip.analysis.coreservices.AnalysisMLKernel kernel pool-zyxqmrmma2jtjakn [20240219-011815.037] [MRT-93702] [INFO] [dku.ml.kernel] - Writing output of python-single-command-kernel to datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1train.log [20240219-011815.037] [MRT-93702] [INFO] [dku.code.envs.resolution] - Executing Python activity in env For_SRI_cutter1 [20240219-011815.038] [MRT-93702] [INFO] [dku.code.projectLibs] - EXTERNAL LIBS FROM LIU_PRACTICE is {gitReferences{},pythonPath[python],rsrcPath[R],importLibrariesFromProjects[]} [20240219-011815.038] [MRT-93702] [DEBUG] [dku.code.projectLibs] - Impersonation enabled, copying project LIU_PRACTICE lib chunk datadataikudss_dataconfigprojectsLIU_PRACTICElibpython to datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1project-python-libsLIU_PRACTICEpython [20240219-011815.038] [MRT-93702] [INFO] [dku.code.projectLibs] - chunkFolder is datadataikudss_dataconfigprojectsLIU_PRACTICElibR [20240219-011815.039] [MRT-93702] [DEBUG] [dku.code.projectLibs] - Impersonation enabled, copying project lib chunk datadataikudss_dataconfigprojectsLIU_PRACTICElibR to datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1project-r-srcR [20240219-011815.039] [MRT-93702] [INFO] [dku.python.single_command.kernel] - Starting Python process for kernel python-single-command-kernel [20240219-011815.039] [MRT-93702] [INFO] [dip.tickets] - Creating API ticket for analysis-ml-LIU_PRACTICE-RseU5GS on behalf of z.ryu@deepblue-ts.co.jp id=analysis-ml-LIU_PRACTICE-RseU5GS_jhhuOYaXpT8I [20240219-011815.039] [MRT-93702] [INFO] [dku.security.fsacl] - chmod datadataikudss_datauif-tmp [OWNER_READ,OWNER_EXECUTE,OTHERS_EXECUTE,OWNER_WRITE,GROUP_EXECUTE] [20240219-011815.039] [MRT-93702] [INFO] [dku.security.process] - Setting exclusive access on process pipes directory datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284 [20240219-011815.039] [MRT-93702] [DEBUG] [dku.security.fsacl] - exec chown usrbinsudo etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJexecwrapper.sh acls chown --path datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284 --owner dssuser_z_ryu_debe3cd638 [20240219-011815.080] [null-err-93706] [INFO] [dku.utils] - 2024-02-19 011815,080 809799 INFO Reading security conf from etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJ [20240219-011815.081] [null-err-93706] [INFO] [dku.utils] - 2024-02-19 011815,081 809799 INFO Allowed path datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284 in datadataikudss_data [20240219-011815.081] [null-err-93706] [INFO] [dku.utils] - 2024-02-19 011815,081 809799 INFO Reading security conf from etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJ [20240219-011815.081] [null-err-93706] [INFO] [dku.utils] - 2024-02-19 011815,081 809799 INFO Auto-create, checking existence of user dssuser_z_ryu_debe3cd638 [20240219-011815.082] [null-err-93706] [INFO] [dku.utils] - 2024-02-19 011815,081 809799 INFO Allowed group ids [1001] [20240219-011815.084] [null-err-93706] [INFO] [dku.utils] - 2024-02-19 011815,084 809799 INFO User dssuser_z_ryu_debe3cd638 belongs to groups [1001] [20240219-011815.085] [null-err-93706] [INFO] [dku.utils] - 2024-02-19 011815,084 809799 INFO User dssuser_z_ryu_debe3cd638 belongs to group 1001 (dss_users) - allowed [20240219-011815.085] [null-err-93706] [INFO] [dku.utils] - 2024-02-19 011815,084 809799 INFO Executing chown -Rh datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284 - dssuser_z_ryu_debe3cd638 [20240219-011815.093] [Thread-45300] [DEBUG] [dku.utils] - Process usrbinsudo done (return code 0) [20240219-011815.093] [MRT-93702] [DEBUG] [dku.security.fsacl] - exec setfacl usrbinsudo etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJexecwrapper.sh acls setfacl --path datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284 --perm urwx --perm g--- --perm o--- --perm dudataikurwx --perm udataikurwx --recursive [20240219-011815.134] [null-err-93709] [INFO] [dku.utils] - 2024-02-19 011815,134 809806 INFO Reading security conf from etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJ [20240219-011815.134] [null-err-93709] [INFO] [dku.utils] - 2024-02-19 011815,134 809806 INFO Allowed path datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284 in datadataikudss_data [20240219-011815.134] [null-err-93709] [INFO] [dku.utils] - 2024-02-19 011815,134 809806 INFO Executing ['setfacl', '-RP', '-m', 'urwx,g---,o---,dudataikurwx,udataikurwx', 'datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284'] [20240219-011815.141] [Thread-45303] [DEBUG] [dku.utils] - Process usrbinsudo done (return code 0) [20240219-011815.142] [MRT-93702] [INFO] [dku.security.process] - Granting access on process working directory datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1 to dssuser_z_ryu_debe3cd638 [20240219-011815.142] [MRT-93702] [DEBUG] [dku.security.fsacl] - exec setfacl usrbinsudo etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJexecwrapper.sh acls setfacl --path datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1 --perm dudssuser_z_ryu_debe3cd638rwx --perm dudataikurwx --perm udssuser_z_ryu_debe3cd638rwx --perm udataikurwx --affected-user dssuser_z_ryu_debe3cd638 --recursive [20240219-011815.182] [null-err-93712] [INFO] [dku.utils] - 2024-02-19 011815,182 809812 INFO Reading security conf from etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJ [20240219-011815.183] [null-err-93712] [INFO] [dku.utils] - 2024-02-19 011815,183 809812 INFO Allowed path datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1 in datadataikudss_data [20240219-011815.183] [null-err-93712] [INFO] [dku.utils] - 2024-02-19 011815,183 809812 INFO Reading security conf from etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJ [20240219-011815.183] [null-err-93712] [INFO] [dku.utils] - 2024-02-19 011815,183 809812 INFO Auto-create, checking existence of user dssuser_z_ryu_debe3cd638 [20240219-011815.183] [null-err-93712] [INFO] [dku.utils] - 2024-02-19 011815,183 809812 INFO Allowed group ids [1001] [20240219-011815.186] [null-err-93712] [INFO] [dku.utils] - 2024-02-19 011815,186 809812 INFO User dssuser_z_ryu_debe3cd638 belongs to groups [1001] [20240219-011815.186] [null-err-93712] [INFO] [dku.utils] - 2024-02-19 011815,186 809812 INFO User dssuser_z_ryu_debe3cd638 belongs to group 1001 (dss_users) - allowed [20240219-011815.186] [null-err-93712] [INFO] [dku.utils] - 2024-02-19 011815,186 809812 INFO Executing ['setfacl', '-RP', '-m', 'dudssuser_z_ryu_debe3cd638rwx,dudataikurwx,udssuser_z_ryu_debe3cd638rwx,udataikurwx', 'datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1'] [20240219-011815.194] [Thread-45306] [DEBUG] [dku.utils] - Process usrbinsudo done (return code 0) [20240219-011815.194] [MRT-93702] [INFO] [dku.processes.cgroups] - cgroup memoryDSS${user}mlKernels expanded to memoryDSSz.ryu@deepblue-ts.co.jpmlKernels [20240219-011815.195] [MRT-93702] [INFO] [dku.processes.cgroups] - Will use cgroups [memoryDSSz.ryu@deepblue-ts.co.jpmlKernels] [20240219-011815.195] [MRT-93702] [INFO] [dku.processes.cgroups] - Applying rules to used cgroups [memoryDSSz.ryu@deepblue-ts.co.jpmlKernels] [20240219-011815.195] [MRT-93702] [INFO] [dku.processes.cgroups] - Rule path memoryDSS [20240219-011815.195] [MRT-93702] [INFO] [dku.processes.cgroups] - Checking match of cgroup setting memoryDSS vs used memoryDSSz.ryu@deepblue-ts.co.jpmlKernels [20240219-011815.195] [MRT-93702] [INFO] [dku.processes.cgroups] - Used cgroupPath memoryDSSz.ryu@deepblue-ts.co.jpmlKernels matches definition for cgroup memoryDSS [20240219-011815.195] [MRT-93702] [INFO] [dku.processes.cgroups] - Setting rules [{keymemory.limit_in_bytes,value45g}] on sysfscgroupmemoryDSS [20240219-011815.195] [MRT-93702] [INFO] [dku.security.process] - Written config to datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284config.json expected SHA256 54da4843c17d15fe772120b2e1e8f0bbdd6b39ddd42a11ea4a16e9a3c68642e7 [20240219-011815.196] [MRT-93702] [INFO] [dku.security.process] - Starting wrapper [usrbinsudo,etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJexecwrapper.sh,execute,-s,54da4843c17d15fe772120b2e1e8f0bbdd6b39ddd42a11ea4a16e9a3c68642e7,datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284config.json] [20240219-011815.244] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,244 809817 INFO Reading security conf from etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJ [20240219-011815.245] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,245 809817 INFO Auto-create, checking existence of user dssuser_z_ryu_debe3cd638 [20240219-011815.245] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,245 809817 INFO Allowed group ids [1001] [20240219-011815.248] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,248 809817 INFO User dssuser_z_ryu_debe3cd638 belongs to groups [1001] [20240219-011815.248] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,248 809817 INFO User dssuser_z_ryu_debe3cd638 belongs to group 1001 (dss_users) - allowed [20240219-011815.248] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,248 809817 INFO Reading security conf from etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJ [20240219-011815.248] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,248 809817 INFO Allowed path datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284 in datadataikudss_data [20240219-011815.248] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,248 809817 INFO Reading security conf from etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJ [20240219-011815.249] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,248 809817 INFO Allowed path datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1 in datadataikudss_data [20240219-011815.249] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,249 809817 INFO [BF] Setting up process environment [20240219-011815.249] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,249 809817 INFO Will set process in cgroup memoryDSSz.ryu@deepblue-ts.co.jpmlKernels [20240219-011815.249] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,249 809817 INFO Reading security conf from etcdataiku-securityyRfRZZ9mjpmtptyTh6HCz1tJ [20240219-011815.249] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,249 809817 INFO Allowed path sysfscgroupmemoryDSSz.ryu@deepblue-ts.co.jpmlKernelstasks in sysfscgroup [20240219-011815.250] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,249 809817 INFO [Parent] Started process 809819 [20240219-011815.250] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,250 809817 INFO [Parent] Writing wrapper pipes and pids to wrapper stdout [20240219-011815.250] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,250 809819 INFO [Child] pgroup is set [20240219-011815.250] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,250 809817 INFO [Parent] Setting process 809819 in cgroup tasks file sysfscgroupmemoryDSSz.ryu@deepblue-ts.co.jpmlKernelstasks [20240219-011815.250] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,250 809819 INFO [Child] opening stdin pipe datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284.stdin.ux4vcjfo [20240219-011815.297] [MRT-93702] [INFO] [dku.security.process] - Reading pids and pipes from wrapper [20240219-011815.297] [MRT-93702] [INFO] [dku.security.process] - Pipes read as bytes 324 [20240219-011815.297] [MRT-93702] [INFO] [dku.security.process] - Pipes read as bytes {pid 809817, pipes {stdin datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284.stdin.ux4vcjfo, stdout datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284.stdout.ux4vcjfo, stderr datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284.stderr.ux4vcjfo}, childPid 809819} [20240219-011815.297] [MRT-93702] [INFO] [dku.security.process] - Pid read 809817 [20240219-011815.297] [MRT-93702] [INFO] [dku.security.process] - Pipes read { stdin datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284.stdin.ux4vcjfo, stdout datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284.stdout.ux4vcjfo, stderr datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284.stderr.ux4vcjfo } [20240219-011815.297] [MRT-93702] [INFO] [dku.security.process] - Child pid read 809819 [20240219-011815.298] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,297 809819 INFO [Child] opened stdin pipe [20240219-011815.298] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,298 809819 INFO [Child] opening stdout pipe datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284.stdout.ux4vcjfo [20240219-011815.298] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,298 809819 INFO [Child] opened stdout [20240219-011815.298] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011815,298 809819 INFO [Child] opening stderr pipe datadataikudss_datauif-tmpLIU_PRACTICE2413543159150645284.stderr.ux4vcjfo [20240219-011815.298] [KNL-python-single-command-kernel-monitor-93715] [DEBUG] [dku.resourceusage] - Reporting start of CRU{context{typeANALYSIS_ML_TRAIN,authIdentifierz.ryu@deepblue-ts.co.jp,projectKeyLIU_PRACTICE,analysisIdHtmoDJNb,mlTaskId2kk5qXQM,sessionIds1},typeLOCAL_PROCESS,id6mh2PU2tNviWHuvZ,startTime1708305495298,localProcess{cpuCurrent0.0,cpuAverageOverPast60Seconds0.0}} 2024-02-19 011815,298 809819 INFO [Child] opened stderr 2024-02-19 011815,298 809819 INFO [Child] about to close other fd 2024-02-19 011815,298 809819 INFO [Child] closed other fd 2024-02-19 011815,298 809819 INFO [Child] chdired 2024-02-19 011815,298 809819 INFO setting username=dssuser_z_ryu_debe3cd638 uid=1012 gid=1001 2024-02-19 011815,299 809819 INFO [Child] dropped privileges 2024-02-19 011815,300 809819 INFO [Child] Checking access to DKUINSTALLDIR and DIP_HOME directories [20240219-011815.300] [process-resource-monitor-809819-93716] [DEBUG] [dku.resource] - Process stats for pid 809819 {pid809819,commandNamedatadataikudss_datacode-envspythonFor_SRI_cutter1binpython,cpuUserTimeMS0,cpuSystemTimeMS0,cpuChildrenUserTimeMS0,cpuChildrenSystemTimeMS0,cpuTotalMS0,cpuCurrent0.0,cpuAverageOverPast60Seconds0.0,vmSizeMB86,vmRSSMB11,vmHWMMB11,vmRSSAnonMB6,vmDataMB6,vmSizePeakMB86,vmRSSPeakMB11,vmRSSTotalMBS0,majorFaults0,childrenMajorFaults0} 2024-02-19 011815,300 809819 INFO [Child] Executing datadataikudss_datacode-envspythonFor_SRI_cutter1binpython datadataikudss_datacode-envspythonFor_SRI_cutter1binpython -u -m dataiku.doctor.server 40459 HogPqqknfbSE5MHN optdataiku-dss-12.0.0pythondataikubaseutils.py214 DeprecationWarning distutils Version classes are deprecated. Use packaging.version instead. return LooseVersion(p.__version__) = LooseVersion(min_version) Installing debugging signal handler [2024-02-19 011817,304] [809819MainThread] [INFO] [dataiku.base.socket_block_link] Connecting to localhost (127.0.0.1) at port 40459 [2024-02-19 011817,304] [809819MainThread] [INFO] [dataiku.base.socket_block_link] Connected to localhost (127.0.0.1) at port 40459 [20240219-011817.305] [MRT-93702] [INFO] [dku.link.secret_protected] - Connected to kernel [20240219-011817.306] [MRT-93702] [INFO] [dku.block.link.interaction] - Execute link command respClazz=true respTypeToken=false respIsString=false is=false asyncInputStream=false os=false [2024-02-19 011817,565] [809819MainThread] [INFO] [root] Running analysis command train_prediction_timeseries [2024-02-19 011817,576] [809819MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback DatasetSanityCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_DATASET_SANITY_CHECKS [2024-02-19 011817,576] [809819MainThread] [INFO] [dataiku.doctor.utils.listener] START - Loading source dataset [2024-02-19 011817,577] [809819MainThread] [INFO] [root] Reading with dtypes None [2024-02-19 011817,577] [809819MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for DateTime None (schema_type=date feature_type=NUMERIC feature_role=TIME) [2024-02-19 011817,577] [809819MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Q class 'numpy.int64' (schema_type=bigint feature_type=NUMERIC feature_role=TIMESERIES_IDENTIFIER) [2024-02-19 011817,577] [809819MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Q1 class 'numpy.float64' (schema_type=bigint feature_type=NUMERIC feature_role=TARGET) [2024-02-19 011817,577] [809819MainThread] [INFO] [root] Reading with FIXED dtypes {'Q' class 'numpy.int64', 'Q1' class 'numpy.float64'} [2024-02-19 011817,599] [809819MainThread] [INFO] [root] Loaded table [2024-02-19 011817,602] [809819MainThread] [INFO] [dataiku.doctor.utils] Normalizing date to datetime and converting to UTC 0 2022-12-21 084524+0000 1 2022-11-17 035925+0000 2 2022-12-21 092507+0000 3 2022-12-21 094321+0000 4 2022-11-17 043257+0000 ... 7072 2023-03-10 012507+0000 7073 2023-03-10 013703+0000 7074 2023-03-10 020113+0000 7075 2023-03-10 021202+0000 7076 2023-03-10 022438+0000 Name DateTime, Length 7077, dtype datetime64[ns, UTC] [2024-02-19 011817,607] [809819MainThread] [INFO] [dataiku.doctor.utils] Normalized date 0 2022-12-21 084524 1 2022-11-17 035925 2 2022-12-21 092507 3 2022-12-21 094321 4 2022-11-17 043257 ... 7072 2023-03-10 012507 7073 2023-03-10 013703 7074 2023-03-10 020113 7075 2023-03-10 021202 7076 2023-03-10 022438 Name DateTime, Length 7077, dtype datetime64[ns] [2024-02-19 011817,607] [809819MainThread] [INFO] [dataiku.doctor.utils] Coercion done [2024-02-19 011817,607] [809819MainThread] [INFO] [dataiku.doctor.utils.split] Loaded train df shape=(7077,3) [2024-02-19 011817,607] [809819MainThread] [INFO] [dataiku.doctor.utils.listener] END - Loading source dataset [20240219-011817.619] [MRT-93702] [INFO] [dku.block.link.interaction] - Check result for nullity exceptionIfNull=true result=null Traceback (most recent call last) File optdataiku-dss-12.0.0pythondataikudoctorserver.py, line 45, in serve ret = api_command(arg) File optdataiku-dss-12.0.0pythondataikudoctordkuapi.py, line 45, in aux return api(kwargs) File optdataiku-dss-12.0.0pythondataikudoctorcommands.py, line 775, in train_prediction_timeseries launch_training(core_params, modeling_sets, preprocessing_params, resampling_params, run_folder, split_desc) File optdataiku-dss-12.0.0pythondataikudoctortimeseriestrainlaunch_training.py, line 58, in launch_training full_df = resample_for_training( File optdataiku-dss-12.0.0pythondataikudoctortimeseriestraintraining_handler.py, line 270, in resample_for_training raise TimeseriesResamplingException(error_message) dataiku.doctor.exception.TimeseriesResamplingException 'Time series (Q 2) cannot be resampled because its target column contains less than 2 valid values.' [2024-02-19 011817,622] [809819MainThread] [INFO] [dataiku.base.socket_block_link] Client closed [20240219-011817.778] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011817,778 809817 INFO [Child] Process 809819 exited with exit=0 signal=0 [20240219-011817.778] [wrapper-stderr-93714] [INFO] [dku.utils] - 2024-02-19 011817,778 809817 INFO Full child code 0 [20240219-011817.785] [KNL-python-single-command-kernel-monitor-93715] [INFO] [dku.kernels] - Process done with code 0 [20240219-011817.786] [KNL-python-single-command-kernel-monitor-93715] [INFO] [dku.engine] - Schedule closing for RemoteWorkerPool [20240219-011817.786] [AsyncCloser-417] [INFO] [dku.ml.distributed.pool] - Closing worker pool pool-zyxqmrmma2jtjakn [20240219-011817.786] [AsyncCloser-417] [INFO] [dku.ml.distributed.service] - Unregistered worker pool pool-zyxqmrmma2jtjakn [20240219-011817.786] [AsyncCloser-417] [INFO] [dku.engine] - Successfully closed RemoteWorkerPool [20240219-011817.786] [KNL-python-single-command-kernel-monitor-93715] [INFO] [dip.tickets] - Destroying API ticket for analysis-ml-LIU_PRACTICE-RseU5GS on behalf of z.ryu@deepblue-ts.co.jp [20240219-011817.786] [KNL-python-single-command-kernel-monitor-93715] [WARN] [dku.resource] - stat file for pid 809819 does not exist. Process died [20240219-011817.787] [KNL-python-single-command-kernel-monitor-93715] [DEBUG] [dku.resourceusage] - Reporting completion of CRU{context{typeANALYSIS_ML_TRAIN,authIdentifierz.ryu@deepblue-ts.co.jp,projectKeyLIU_PRACTICE,analysisIdHtmoDJNb,mlTaskId2kk5qXQM,sessionIds1},typeLOCAL_PROCESS,id6mh2PU2tNviWHuvZ,startTime1708305495298,localProcess{pid809819,commandNamedatadataikudss_datacode-envspythonFor_SRI_cutter1binpython,cpuUserTimeMS0,cpuSystemTimeMS0,cpuChildrenUserTimeMS0,cpuChildrenSystemTimeMS0,cpuTotalMS0,cpuCurrent0.0,cpuAverageOverPast60Seconds0.0,vmSizeMB86,vmRSSMB11,vmHWMMB11,vmRSSAnonMB6,vmDataMB6,vmSizePeakMB86,vmRSSPeakMB11,vmRSSTotalMBS0,majorFaults0,childrenMajorFaults0}} [20240219-011817.787] [MRT-93702] [INFO] [dku.kernels] - Getting kernel tail [20240219-011817.788] [MRT-93702] [INFO] [dku.kernels] - Process was cleaned up by monitoring thread [20240219-011817.789] [MRT-93702] [INFO] [dku.kernels] - Trying to enrich exception com.dataiku.dip.io.SocketBlockLinkKernelException Failed to train class 'dataiku.doctor.exception.TimeseriesResamplingException' Time series (Q 2) cannot be resampled because its target column contains less than 2 valid values. from kernel com.dataiku.dip.analysis.coreservices.AnalysisMLKernel@42de6720 retcode=0 [20240219-011817.789] [MRT-93702] [WARN] [dku.analysis.ml.python] - Training failed com.dataiku.dip.io.SocketBlockLinkKernelException Failed to train class 'dataiku.doctor.exception.TimeseriesResamplingException' Time series (Q 2) cannot be resampled because its target column contains less than 2 valid values. at com.dataiku.dip.io.SocketBlockLinkInteraction.throwExceptionFromPython(SocketBlockLinkInteraction.java302) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.checkException(SocketBlockLinkInteraction.java215) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.get(SocketBlockLinkInteraction.java190) at com.dataiku.dip.io.SingleCommandKernelLink$1.call(SingleCommandKernelLink.java211) at com.dataiku.dip.analysis.ml.prediction.PredictionTrainAdditionalThread.process(PredictionTrainAdditionalThread.java76) at com.dataiku.dip.analysis.ml.shared.PRNSTrainThread.run(PRNSTrainThread.java172) [20240219-011817.789] [MRT-93702] [INFO] [dku.block.link] - Closed socket [20240219-011817.789] [MRT-93702] [INFO] [dku.block.link] - Closed socket [20240219-011817.789] [MRT-93702] [INFO] [dku.block.link] - Closed serverSocket [20240219-011817.789] [MRT-93702] [ERROR] [dku.analysis.ml.python] - Processing failed com.dataiku.dip.io.SocketBlockLinkKernelException Failed to train class 'dataiku.doctor.exception.TimeseriesResamplingException' Time series (Q 2) cannot be resampled because its target column contains less than 2 valid values. at com.dataiku.dip.io.SocketBlockLinkInteraction.throwExceptionFromPython(SocketBlockLinkInteraction.java302) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.checkException(SocketBlockLinkInteraction.java215) at com.dataiku.dip.io.SocketBlockLinkInteraction$AsyncResult.get(SocketBlockLinkInteraction.java190) at com.dataiku.dip.io.SingleCommandKernelLink$1.call(SingleCommandKernelLink.java211) at com.dataiku.dip.analysis.ml.prediction.PredictionTrainAdditionalThread.process(PredictionTrainAdditionalThread.java76) at com.dataiku.dip.analysis.ml.shared.PRNSTrainThread.run(PRNSTrainThread.java172) [20240219-011817.790] [MRT-93702] [INFO] [dku.analysis.ml] - Locking model train info file datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1m1train_info.json [20240219-011817.791] [MRT-93702] [INFO] [dku.analysis.ml] - Unlocking model train info file datadataikudss_dataanalysis-dataLIU_PRACTICEHtmoDJNb2kk5qXQMsessionss1pp1m1train_info.json [20240219-011817.791] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.ml.python] T-2kk5qXQM - [ct 3148] Processing thread joined ... [20240219-011817.791] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.ml.python] T-2kk5qXQM - [ct 3148] Joining processing thread ... [20240219-011817.791] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.ml.python] T-2kk5qXQM - [ct 3148] Processing thread joined ... [20240219-011817.791] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis] T-2kk5qXQM - [ct 3148] Train done [20240219-011817.791] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.prediction] T-2kk5qXQM - [ct 3148] Train done [20240219-011817.793] [FT-TrainWorkThread-GjmjUROB-93700] [INFO] [dku.analysis.trainingdetails] T-2kk5qXQM - Publishing mltask-train-done reflected event