DSS backend failed to start, project file corrupted

Mathieu7

Hello,

Hi, im running dataiku on ubuntu/WIndows

Help

dss has been fine until today when it froze while being used to creating a new dataset, i stopped dss and rebooted and since then wont restart

dss status says

backend FATAL exited too quickly

backend log in the last entries keep referring to the recipe json file is corrupted and this stop the whole dss from starting

lots of other lines in the backend.log file about BEAN creation exceptions (def out of my depth on those ones!)

thanks

backend most recent:

[2022-10-05T10:22:27.244+0200][32ms][1832][info][gc] Using G1
[2022-10-05T10:22:28.380+0200][1168ms][1838][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 40M->29M(256M) 43.200ms
[2022-10-05T10:22:28.380+0200][1168ms][1838][info][gc,cpu] GC(0) User=0.06s Sys=0.07s Real=0.04s
[2022/10/05-10:22:28.395] [main] [INFO] [dku.logging] - Loading logging settings
[2022/10/05-10:22:28.403] [main] [INFO] [dku.logging] - Configuring additional logging settings from /home/dataiku/dataiku-dss-10.0.2/resources/logging/dku-log4j.properties
[2022/10/05-10:22:28.437] [main] [INFO] [dku.logging] - Configuring additional JUL logging settings from /home/dataiku/dataiku-dss-10.0.2/resources/logging/dku-log-jul.properties
[2022/10/05-10:22:28.543] [main] [INFO] [dku] - ----------------------------------------
[2022/10/05-10:22:28.543] [main] [INFO] [dku] - DSS startup: backend version:10.0.2
[2022/10/05-10:22:28.543] [main] [INFO] [dku] - DSS home: /home/dataiku/DATA_DIR
[2022/10/05-10:22:28.543] [main] [INFO] [dku] - OS: Linux 4.4.0-19041-Microsoft amd64 - Java: Ubuntu 11.0.13
[2022-10-05T10:22:28.830+0200][1618ms][1838][info][gc ] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 44M->29M(256M) 9.757ms
[2022-10-05T10:22:28.831+0200][1619ms][1838][info][gc,cpu] GC(1) User=0.05s Sys=0.01s Real=0.01s
[2022/10/05-10:22:29.575] [main] [INFO] [dku.builtins] - Loading countries geo data
[2022/10/05-10:22:29.612] [main] [INFO] [dku] - Starting on port 11001
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/dataiku/dataiku-dss-10.0.2/lib/ivy/backend-run/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/dataiku/dataiku-dss-10.0.2/lib/ivy/common-run/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2022/10/05-10:22:29.673] [main] [INFO] [org.eclipse.jetty.util.log] - Logging initialized @2457ms to org.eclipse.jetty.util.log.Slf4jLog
[2022/10/05-10:22:29.721] [main] [INFO] [com.dataiku.dip.DKUApp] - Reading /home/dataiku/DATA_DIR/install.ini
[2022-10-05T10:22:29.756+0200][2544ms][1838][info][gc ] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 63M->31M(256M) 4.437ms
[2022-10-05T10:22:29.756+0200][2544ms][1838][info][gc,cpu] GC(2) User=0.00s Sys=0.00s Real=0.01s
[2022/10/05-10:22:30.196] [main] [INFO] [org.eclipse.jetty.server.Server] - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 11.0.13+8-Ubuntu-0ubuntu1.18.04
[2022/10/05-10:22:30.496] [main] [INFO] [org.eclipse.jetty.webapp.StandardDescriptorProcessor] - NO JSP Support for /dip, did not find org.eclipse.jetty.jsp.JettyJspServlet
[2022-10-05T10:22:30.516+0200][3304ms][1838][info][gc ] GC(3) Pause Young (Concurrent Start) (Metadata GC Threshold) 66M->33M(256M) 17.008ms
[2022-10-05T10:22:30.516+0200][3304ms][1838][info][gc,cpu] GC(3) User=0.02s Sys=0.05s Real=0.02s
[2022-10-05T10:22:30.516+0200][3304ms][1834][info][gc ] GC(4) Concurrent Cycle
[2022-10-05T10:22:30.531+0200][3319ms][1838][info][gc ] GC(4) Pause Remark 33M->33M(256M) 3.920ms
[2022-10-05T10:22:30.531+0200][3319ms][1838][info][gc,cpu] GC(4) User=0.00s Sys=0.00s Real=0.00s
[2022-10-05T10:22:30.535+0200][3323ms][1838][info][gc ] GC(4) Pause Cleanup 33M->33M(256M) 0.180ms
[2022-10-05T10:22:30.535+0200][3323ms][1838][info][gc,cpu] GC(4) User=0.00s Sys=0.00s Real=0.00s
[2022-10-05T10:22:30.541+0200][3329ms][1834][info][gc ] GC(4) Concurrent Cycle 25.317ms
[2022/10/05-10:22:30.582] [main] [INFO] [org.eclipse.jetty.server.session] - DefaultSessionIdManager workerName=node0
[2022/10/05-10:22:30.582] [main] [INFO] [org.eclipse.jetty.server.session] - No SessionScavenger set, using defaults
[2022/10/05-10:22:30.588] [main] [INFO] [org.eclipse.jetty.server.session] - node0 Scavenging every 600000ms
[+] Dataiku DSS starting up (DIP_HOME=/home/dataiku/DATA_DIR)
[2022/10/05-10:22:30.616] [main] [INFO] [org.eclipse.jetty.server.handler.ContextHandler.dip] - Initializing Spring root WebApplicationContext
[2022/10/05-10:22:30.776] [main] [INFO] [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - Loading XML bean definitions from ServletContext resource [/WEB-INF/applicationContext.xml]
[2022-10-05T10:22:31.362+0200][4150ms][1838][info][gc ] GC(5) Pause Young (Normal) (G1 Evacuation Pause) 94M->35M(256M) 23.581ms
[2022-10-05T10:22:31.363+0200][4151ms][1838][info][gc,cpu] GC(5) User=0.03s Sys=0.04s Real=0.02s
[2022-10-05T10:22:32.085+0200][4873ms][1838][info][gc ] GC(6) Pause Young (Normal) (G1 Evacuation Pause) 126M->41M(345M) 31.735ms
[2022-10-05T10:22:32.085+0200][4873ms][1838][info][gc,cpu] GC(6) User=0.14s Sys=0.09s Real=0.04s
[2022-10-05T10:22:33.460+0200][6248ms][1838][info][gc ] GC(7) Pause Young (Normal) (G1 Evacuation Pause) 162M->44M(345M) 21.644ms
[2022-10-05T10:22:33.460+0200][6248ms][1838][info][gc,cpu] GC(7) User=0.10s Sys=0.02s Real=0.02s
[2022-10-05T10:22:36.023+0200][8811ms][1838][info][gc ] GC(8) Pause Young (Normal) (G1 Evacuation Pause) 242M->45M(345M) 338.410ms
[2022-10-05T10:22:36.030+0200][8818ms][1838][info][gc,cpu] GC(8) User=0.42s Sys=0.04s Real=0.33s
[2022-10-05T10:22:38.753+0200][11541ms][1838][info][gc ] GC(9) Pause Young (Normal) (G1 Evacuation Pause) 242M->42M(345M) 25.797ms
[2022-10-05T10:22:38.753+0200][11541ms][1838][info][gc,cpu] GC(9) User=0.03s Sys=0.00s Real=0.02s
[2022/10/05-10:22:39.650] [main] [INFO] [dku.transactions.service] - Instantiating backend transaction service
[2022/10/05-10:22:39.673] [main] [WARN] [dip.transactions] - The file /home/dataiku/DATA_DIR/config/.ts/projects--TERRITORYQUALITYAUDIT--recipes--split_OCB_DATA.json doesn't exist. If you are recovering from a failure, it may be normal. Otherwise, this is likely to be a critical issue.
[2022/10/05-10:22:39.683] [main] [ERROR] [dip.transactions] - Here is the current content of /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is (md5=7faf1375fed8b4d653a2e682683a5d33):
{
"type": "split",
"neverRecomputeExistingPartitions": false,
"optionalDependencies": false,
"customMeta": {
"kv": {}
},
"redispatchPartitioning": false,
"maxRunningActivities": 0,
"variables": {},
"dkuProperties": [],
"inputs": {
"main": {
"items": [
{
"ref": "OCB_DATA_prepared",
"deps": []
}
]
}
},
"outputs": {
"main": {
"items": [
{
"ref": "OCB_NON_Integrated_Legacy_Brand",
"appendMode": false
},
{
"ref": "OCB_Integrated_Legacy_Brand",
"appendMode": false
}
]
}
},
"labels": [],
"versionTag": {
"versionNumber": 2,
"lastModifiedBy": {
"login": "admin"
},
"lastModifiedOn": 1664805746047
},
"creationTag": {
"versionNumber": 0,
"lastModifiedBy": {
"login": "admin"
},
"lastModifiedOn": 1664805407965
},
"tags": [],
"customFields": {},
"checklists": {
"checklists": []
}
}
[2022/10/05-10:22:39.683] [main] [ERROR] [dip.transactions] - The expected content of /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is not available anymore because DSS has been restarted.
[2022/10/05-10:22:39.687] [main] [WARN] [org.springframework.web.context.support.XmlWebApplicationContext] - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendComputeResourceUsageReportingService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.dataiku.dip.security.audit.AuditTrailService com.dataiku.dip.resourceusage.BackendComputeResourceUsageReportingService.auditTrailService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'auditTrailService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.security.audit.AuditTrailSettingsProvider com.dataiku.dip.security.audit.AuditTrailService.settingsProvider; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DSSAuditTrailSettingsProvider': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.dao.GeneralSettingsDAO com.dataiku.dip.dao.DSSAuditTrailSettingsProvider.generalSettingsDAO; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'generalSettingsDAO': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.server.services.TransactionService com.dataiku.dip.dao.GeneralSettingsDAO.transactionService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendTransactionService' defined in URL [jar:file:/home/dataiku/dataiku-dss-10.0.2/dist/dataiku-dip.jar!/com/dataiku/dip/server/services/BackendTransactionService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.dataiku.dip.server.services.BackendTransactionService]: Constructor threw exception; nested exception is java.io.IOError: java.io.IOException: The file /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is corrupted
[2022/10/05-10:22:39.689] [main] [ERROR] [org.springframework.web.context.ContextLoader] - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendComputeResourceUsageReportingService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.dataiku.dip.security.audit.AuditTrailService com.dataiku.dip.resourceusage.BackendComputeResourceUsageReportingService.auditTrailService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'auditTrailService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.security.audit.AuditTrailSettingsProvider com.dataiku.dip.security.audit.AuditTrailService.settingsProvider; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DSSAuditTrailSettingsProvider': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.dao.GeneralSettingsDAO com.dataiku.dip.dao.DSSAuditTrailSettingsProvider.generalSettingsDAO; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'generalSettingsDAO': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.server.services.TransactionService com.dataiku.dip.dao.GeneralSettingsDAO.transactionService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendTransactionService' defined in URL [jar:file:/home/dataiku/dataiku-dss-10.0.2/dist/dataiku-dip.jar!/com/dataiku/dip/server/services/BackendTransactionService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.dataiku.dip.server.services.BackendTransactionService]: Constructor threw exception; nested exception is java.io.IOError: java.io.IOException: The file /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is corrupted
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:638)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at com.dataiku.dip.DSSBackendMain$ThrowyWebAppContext.doStart(DSSBackendMain.java:167)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:408)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at com.dataiku.dip.DSSBackendMain.main(DSSBackendMain.java:147)
Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.dataiku.dip.security.audit.AuditTrailService com.dataiku.dip.resourceusage.BackendComputeResourceUsageReportingService.auditTrailService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'auditTrailService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.security.audit.AuditTrailSettingsProvider com.dataiku.dip.security.audit.AuditTrailService.settingsProvider; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DSSAuditTrailSettingsProvider': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.dao.GeneralSettingsDAO com.dataiku.dip.dao.DSSAuditTrailSettingsProvider.generalSettingsDAO; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'generalSettingsDAO': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.server.services.TransactionService com.dataiku.dip.dao.GeneralSettingsDAO.transactionService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendTransactionService' defined in URL [jar:file:/home/dataiku/dataiku-dss-10.0.2/dist/dataiku-dip.jar!/com/dataiku/dip/server/services/BackendTransactionService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.dataiku.dip.server.services.BackendTransactionService]: Constructor threw exception; nested exception is java.io.IOError: java.io.IOException: The file /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is corrupted
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:531)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:295)
... 32 more

tgb417

@Mathieu7
,

Welcome to the Dataiku Community. We are so glad to have you join us here.

Given the nature of your problem, I'd suggest that you submit a support ticket.

You can get to the place to create support tickets in Data Science Studio (DSS) by going to the upper right-hand corner of the DSS web page and choosing the question mark in the circle icon (?)

There choose the "get help" item.

Then choose "open ticket" option.

Alexandru

HI @Mathieu7
,

Indeed this is a support issue as @tgb417
suggested. But please note we do not support running DSS on WLS( Windows Subsystem for Linux) ( Ubuntu) . While it can work at times it's untested and unsupported.

The error appears to originate from some corrupted files :
/home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json

I would suggest moving /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT outside of the DATA_DIR and then restarting DSS if that works and you can easily recreate the projects you should be fine. Otherwise, please open a support ticket.