DSS backend failed to start, project file corrupted

Hello,

Hi, im running dataiku on ubuntu/WIndows

Help

dss has been fine until today when it froze while being used to creating a new dataset, i stopped dss and rebooted and since then wont restart

dss status says

backend FATAL exited too quickly

backend log in the last entries keep referring to the recipe json file is corrupted and this stop the whole dss from starting

lots of other lines in the backend.log file about BEAN creation exceptions (def out of my depth on those ones!)

thanks

backend most recent:

[2022-10-05T10:22:27.244+0200][32ms][1832][info][gc] Using G1
[2022-10-05T10:22:28.380+0200][1168ms][1838][info][gc] GC(0) Pause Young (Normal) (G1 Evacuation Pause) 40M->29M(256M) 43.200ms
[2022-10-05T10:22:28.380+0200][1168ms][1838][info][gc,cpu] GC(0) User=0.06s Sys=0.07s Real=0.04s
[2022/10/05-10:22:28.395] [main] [INFO] [dku.logging] - Loading logging settings
[2022/10/05-10:22:28.403] [main] [INFO] [dku.logging] - Configuring additional logging settings from /home/dataiku/dataiku-dss-10.0.2/resources/logging/dku-log4j.properties
[2022/10/05-10:22:28.437] [main] [INFO] [dku.logging] - Configuring additional JUL logging settings from /home/dataiku/dataiku-dss-10.0.2/resources/logging/dku-log-jul.properties
[2022/10/05-10:22:28.543] [main] [INFO] [dku] - ----------------------------------------
[2022/10/05-10:22:28.543] [main] [INFO] [dku] - DSS startup: backend version:10.0.2
[2022/10/05-10:22:28.543] [main] [INFO] [dku] - DSS home: /home/dataiku/DATA_DIR
[2022/10/05-10:22:28.543] [main] [INFO] [dku] - OS: Linux 4.4.0-19041-Microsoft amd64 - Java: Ubuntu 11.0.13
[2022-10-05T10:22:28.830+0200][1618ms][1838][info][gc ] GC(1) Pause Young (Normal) (G1 Evacuation Pause) 44M->29M(256M) 9.757ms
[2022-10-05T10:22:28.831+0200][1619ms][1838][info][gc,cpu] GC(1) User=0.05s Sys=0.01s Real=0.01s
[2022/10/05-10:22:29.575] [main] [INFO] [dku.builtins] - Loading countries geo data
[2022/10/05-10:22:29.612] [main] [INFO] [dku] - Starting on port 11001
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/dataiku/dataiku-dss-10.0.2/lib/ivy/backend-run/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/dataiku/dataiku-dss-10.0.2/lib/ivy/common-run/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2022/10/05-10:22:29.673] [main] [INFO] [org.eclipse.jetty.util.log] - Logging initialized @2457ms to org.eclipse.jetty.util.log.Slf4jLog
[2022/10/05-10:22:29.721] [main] [INFO] [com.dataiku.dip.DKUApp] - Reading /home/dataiku/DATA_DIR/install.ini
[2022-10-05T10:22:29.756+0200][2544ms][1838][info][gc ] GC(2) Pause Young (Normal) (G1 Evacuation Pause) 63M->31M(256M) 4.437ms
[2022-10-05T10:22:29.756+0200][2544ms][1838][info][gc,cpu] GC(2) User=0.00s Sys=0.00s Real=0.01s
[2022/10/05-10:22:30.196] [main] [INFO] [org.eclipse.jetty.server.Server] - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 11.0.13+8-Ubuntu-0ubuntu1.18.04
[2022/10/05-10:22:30.496] [main] [INFO] [org.eclipse.jetty.webapp.StandardDescriptorProcessor] - NO JSP Support for /dip, did not find org.eclipse.jetty.jsp.JettyJspServlet
[2022-10-05T10:22:30.516+0200][3304ms][1838][info][gc ] GC(3) Pause Young (Concurrent Start) (Metadata GC Threshold) 66M->33M(256M) 17.008ms
[2022-10-05T10:22:30.516+0200][3304ms][1838][info][gc,cpu] GC(3) User=0.02s Sys=0.05s Real=0.02s
[2022-10-05T10:22:30.516+0200][3304ms][1834][info][gc ] GC(4) Concurrent Cycle
[2022-10-05T10:22:30.531+0200][3319ms][1838][info][gc ] GC(4) Pause Remark 33M->33M(256M) 3.920ms
[2022-10-05T10:22:30.531+0200][3319ms][1838][info][gc,cpu] GC(4) User=0.00s Sys=0.00s Real=0.00s
[2022-10-05T10:22:30.535+0200][3323ms][1838][info][gc ] GC(4) Pause Cleanup 33M->33M(256M) 0.180ms
[2022-10-05T10:22:30.535+0200][3323ms][1838][info][gc,cpu] GC(4) User=0.00s Sys=0.00s Real=0.00s
[2022-10-05T10:22:30.541+0200][3329ms][1834][info][gc ] GC(4) Concurrent Cycle 25.317ms
[2022/10/05-10:22:30.582] [main] [INFO] [org.eclipse.jetty.server.session] - DefaultSessionIdManager workerName=node0
[2022/10/05-10:22:30.582] [main] [INFO] [org.eclipse.jetty.server.session] - No SessionScavenger set, using defaults
[2022/10/05-10:22:30.588] [main] [INFO] [org.eclipse.jetty.server.session] - node0 Scavenging every 600000ms
[+] Dataiku DSS starting up (DIP_HOME=/home/dataiku/DATA_DIR)
[2022/10/05-10:22:30.616] [main] [INFO] [org.eclipse.jetty.server.handler.ContextHandler.dip] - Initializing Spring root WebApplicationContext
[2022/10/05-10:22:30.776] [main] [INFO] [org.springframework.beans.factory.xml.XmlBeanDefinitionReader] - Loading XML bean definitions from ServletContext resource [/WEB-INF/applicationContext.xml]
[2022-10-05T10:22:31.362+0200][4150ms][1838][info][gc ] GC(5) Pause Young (Normal) (G1 Evacuation Pause) 94M->35M(256M) 23.581ms
[2022-10-05T10:22:31.363+0200][4151ms][1838][info][gc,cpu] GC(5) User=0.03s Sys=0.04s Real=0.02s
[2022-10-05T10:22:32.085+0200][4873ms][1838][info][gc ] GC(6) Pause Young (Normal) (G1 Evacuation Pause) 126M->41M(345M) 31.735ms
[2022-10-05T10:22:32.085+0200][4873ms][1838][info][gc,cpu] GC(6) User=0.14s Sys=0.09s Real=0.04s
[2022-10-05T10:22:33.460+0200][6248ms][1838][info][gc ] GC(7) Pause Young (Normal) (G1 Evacuation Pause) 162M->44M(345M) 21.644ms
[2022-10-05T10:22:33.460+0200][6248ms][1838][info][gc,cpu] GC(7) User=0.10s Sys=0.02s Real=0.02s
[2022-10-05T10:22:36.023+0200][8811ms][1838][info][gc ] GC(8) Pause Young (Normal) (G1 Evacuation Pause) 242M->45M(345M) 338.410ms
[2022-10-05T10:22:36.030+0200][8818ms][1838][info][gc,cpu] GC(8) User=0.42s Sys=0.04s Real=0.33s
[2022-10-05T10:22:38.753+0200][11541ms][1838][info][gc ] GC(9) Pause Young (Normal) (G1 Evacuation Pause) 242M->42M(345M) 25.797ms
[2022-10-05T10:22:38.753+0200][11541ms][1838][info][gc,cpu] GC(9) User=0.03s Sys=0.00s Real=0.02s
[2022/10/05-10:22:39.650] [main] [INFO] [dku.transactions.service] - Instantiating backend transaction service
[2022/10/05-10:22:39.673] [main] [WARN] [dip.transactions] - The file /home/dataiku/DATA_DIR/config/.ts/projects--TERRITORYQUALITYAUDIT--recipes--split_OCB_DATA.json doesn't exist. If you are recovering from a failure, it may be normal. Otherwise, this is likely to be a critical issue.
[2022/10/05-10:22:39.683] [main] [ERROR] [dip.transactions] - Here is the current content of /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is (md5=7faf1375fed8b4d653a2e682683a5d33):
{
"type": "split",
"neverRecomputeExistingPartitions": false,
"optionalDependencies": false,
"customMeta": {
"kv": {}
},
"redispatchPartitioning": false,
"maxRunningActivities": 0,
"variables": {},
"dkuProperties": [],
"inputs": {
"main": {
"items": [
{
"ref": "OCB_DATA_prepared",
"deps": []
}
]
}
},
"outputs": {
"main": {
"items": [
{
"ref": "OCB_NON_Integrated_Legacy_Brand",
"appendMode": false
},
{
"ref": "OCB_Integrated_Legacy_Brand",
"appendMode": false
}
]
}
},
"labels": [],
"versionTag": {
"versionNumber": 2,
"lastModifiedBy": {
"login": "admin"
},
"lastModifiedOn": 1664805746047
},
"creationTag": {
"versionNumber": 0,
"lastModifiedBy": {
"login": "admin"
},
"lastModifiedOn": 1664805407965
},
"tags": [],
"customFields": {},
"checklists": {
"checklists": []
}
}
[2022/10/05-10:22:39.683] [main] [ERROR] [dip.transactions] - The expected content of /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is not available anymore because DSS has been restarted.
[2022/10/05-10:22:39.687] [main] [WARN] [org.springframework.web.context.support.XmlWebApplicationContext] - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendComputeResourceUsageReportingService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.dataiku.dip.security.audit.AuditTrailService com.dataiku.dip.resourceusage.BackendComputeResourceUsageReportingService.auditTrailService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'auditTrailService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.security.audit.AuditTrailSettingsProvider com.dataiku.dip.security.audit.AuditTrailService.settingsProvider; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DSSAuditTrailSettingsProvider': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.dao.GeneralSettingsDAO com.dataiku.dip.dao.DSSAuditTrailSettingsProvider.generalSettingsDAO; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'generalSettingsDAO': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.server.services.TransactionService com.dataiku.dip.dao.GeneralSettingsDAO.transactionService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendTransactionService' defined in URL [jar:file:/home/dataiku/dataiku-dss-10.0.2/dist/dataiku-dip.jar!/com/dataiku/dip/server/services/BackendTransactionService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.dataiku.dip.server.services.BackendTransactionService]: Constructor threw exception; nested exception is java.io.IOError: java.io.IOException: The file /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is corrupted
[2022/10/05-10:22:39.689] [main] [ERROR] [org.springframework.web.context.ContextLoader] - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendComputeResourceUsageReportingService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.dataiku.dip.security.audit.AuditTrailService com.dataiku.dip.resourceusage.BackendComputeResourceUsageReportingService.auditTrailService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'auditTrailService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.security.audit.AuditTrailSettingsProvider com.dataiku.dip.security.audit.AuditTrailService.settingsProvider; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DSSAuditTrailSettingsProvider': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.dao.GeneralSettingsDAO com.dataiku.dip.dao.DSSAuditTrailSettingsProvider.generalSettingsDAO; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'generalSettingsDAO': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.server.services.TransactionService com.dataiku.dip.dao.GeneralSettingsDAO.transactionService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendTransactionService' defined in URL [jar:file:/home/dataiku/dataiku-dss-10.0.2/dist/dataiku-dip.jar!/com/dataiku/dip/server/services/BackendTransactionService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.dataiku.dip.server.services.BackendTransactionService]: Constructor threw exception; nested exception is java.io.IOError: java.io.IOException: The file /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is corrupted
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:638)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1013)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:942)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:782)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at com.dataiku.dip.DSSBackendMain$ThrowyWebAppContext.doStart(DSSBackendMain.java:167)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:408)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at com.dataiku.dip.DSSBackendMain.main(DSSBackendMain.java:147)
Caused by: org.springframework.beans.factory.BeanCreationException: Could not autowire field: com.dataiku.dip.security.audit.AuditTrailService com.dataiku.dip.resourceusage.BackendComputeResourceUsageReportingService.auditTrailService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'auditTrailService': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.security.audit.AuditTrailSettingsProvider com.dataiku.dip.security.audit.AuditTrailService.settingsProvider; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'DSSAuditTrailSettingsProvider': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.dao.GeneralSettingsDAO com.dataiku.dip.dao.DSSAuditTrailSettingsProvider.generalSettingsDAO; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'generalSettingsDAO': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.dataiku.dip.server.services.TransactionService com.dataiku.dip.dao.GeneralSettingsDAO.transactionService; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'backendTransactionService' defined in URL [jar:file:/home/dataiku/dataiku-dss-10.0.2/dist/dataiku-dip.jar!/com/dataiku/dip/server/services/BackendTransactionService.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.dataiku.dip.server.services.BackendTransactionService]: Constructor threw exception; nested exception is java.io.IOError: java.io.IOException: The file /home/dataiku/DATA_DIR/config/projects/TERRITORYQUALITYAUDIT/recipes/split_OCB_DATA.json is corrupted
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:531)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:295)
... 32 more