Log nested exceptions

natejgardner

It seems that when running jobs, if a nested exception occurs, only the top level exception is logged. This makes debugging very difficult, since the true error is not visible. Many Teradata errors occur as nested exceptions, so it would be very helpful if the Dataiku logger could tree down to the complete exception and write that to the log so the underlying issue behind these errors can be understood. I've gotten these messages for everything from out of space to invalid characters to overflowing a field. While I usually know to look for these conditions manually, it's very time consuming. Unfortunately, this sort of error also usually happens after a long execution time. For example, this one took 56 minutes to error out. That's a long iteration loop for trial and error processes, so a little more info here would save me hours. Example:

[14:48:05] [ERROR] [dku.output.sql] - Nested exception
java.sql.SQLException: [Teradata JDBC Driver] [TeraJDBC 16.20.00.12] [Error 1339] [SQLState HY000] A failure occurred while executing a PreparedStatement batch request. The parameter set was not executed and should be resubmitted individually using the PreparedStatement executeUpdate method.
   at com.teradata.jdbc.jdbc_4.util.ErrorFactory.cloneJDBCException(ErrorFactory.java:171)
   at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeChainedBatchJDBCException(ErrorFactory.java:164)
   at com.teradata.jdbc.jdbc_4.statemachine.PreparedBatchStatementController.handleRunException(PreparedBatchStatementController.java:98)
   at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:145)
   at com.teradata.jdbc.jdbc_4.statemachine.PreparedBatchStatementController.run(PreparedBatchStatementController.java:58)
   at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:389)
   at com.teradata.jdbc.jdbc_4.TDPreparedStatement.executeBatchDMLArray(TDPreparedStatement.java:256)
   at com.teradata.jdbc.jdbc_4.TDPreparedStatement.executeBatch(TDPreparedStatement.java:2754)
   at com.dataiku.dip.datasets.sql.SQLTableOutput$SQLTableOutputWriter.emitRow(SQLTableOutput.java:230)
   at com.dataiku.dip.dataflow.exec.stream.ToDatasetStreamer$Streamer.processRow(ToDatasetStreamer.java:131)
   at com.dataiku.dip.datalayer.ProcessorOutputToSIP.emitRow(ProcessorOutputToSIP.java:24)
   at com.dataiku.dip.recipes.code.sql.SQLQueryRecipeUtils.streamSingleRow(SQLQueryRecipeUtils.java:296)
   at com.dataiku.dip.recipes.code.sql.AbstractSQLQueryRecipeRunner.runRegularToDataset(AbstractSQLQueryRecipeRunner.java:225)
   at com.dataiku.dip.dataflow.exec.sql.SQLQueryRecipeRunner.runRegular(SQLQueryRecipeRunner.java:223)
   at com.dataiku.dip.dataflow.exec.sql.SQLQueryRecipeRunner.run(SQLQueryRecipeRunner.java:164)
   at com.dataiku.dip.dataflow.exec.MultiEngineRecipeRunner.run(MultiEngineRecipeRunner.java:203)
   at com.dataiku.dip.dataflow.jobrunner.ActivityRunner$FlowRunnableThread.run(ActivityRunner.java:380)

natejgardner

Just following up on this, it'd really help quality of life for Teradata users to have this. Basically, there's very little error logging for Teradata right now, since the majority of errors are nested exceptions. It's very difficult to troubleshoot failing recipes without errors being logged.