Tried running manual scenario but fails at AssertionError

casper

I have no idea how to solve this.

I've got two projects that share some datasets. For simplicity my summarized problem:

I have a small pipeline in project A, call it Teams. I've shared this dataset, the default CSV/Dataiku file based dataset, with project B, call it Projects.

I've created a manual scenario that first asks project A to forced recursive build the 'teams_available' dataset so that next I can build the dataset "projects_assigned" in project B using the shared dataset.

This fails as easy with 2 steps for no apparent reason but an assertion error.

See:

With the relevant log only being:

[2019/02/14-15:14:52.425] [qtp1914740115-24] [ERROR] [dku.jobs]  - Call failed
java.lang.AssertionError
 at com.dataiku.dip.dataflow.JobComputer.compute(JobComputer.java:111)
 at com.dataiku.dip.dataflow.kernel.slave.KernelSession.resolve(KernelSession.java:184)
    at com.dataiku.dip.dataflow.kernel.slave.KernelServlet.service(KernelServlet.java:163)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
   at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
 at org.eclipse.jetty.server.Server.handle(Server.java:462)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
  at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
   at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
    at java.lang.Thread.run(Thread.java:748)
[2019/02/14-15:14:52.440] [qtp1914740115-24] [DEBUG] [dku.jobs]  - Command /pintercom/resolve_job processed in 314ms
2019-02-14T15:14:52.447: Unexpected ERROR waiting for job to complete
com.dataiku.common.server.APIError$SerializedErrorException: Internal error, caused by: AssertionError: null
   at com.dataiku.dip.dataflow.kernel.master.JobExecutionKernelHandle.executeCommand(JobExecutionKernelHandle.java:277)
  at com.dataiku.dip.dataflow.kernel.master.JobExecutionKernelHandle.executeCommand(JobExecutionKernelHandle.java:251)
  at com.dataiku.dip.dataflow.kernel.master.BuildService$CombinedExecWaitThread.run(BuildService.java:492)

Leaving me no hints or whatsoever. What could cause this problem?

Clément_Stenac

Hi,

This error means that the project the job is requested in doesn't exist. Could you check if the project key you're using is correct?

casper

That was fast! Thanks for coming back to me this fast!

I am not sure what you mean because I am never asked for a proejct key at all. I just use the GUI and add scenario steps, "Build / Train", Add Dataset and select the correct dataset from the correct project. There is no mistaking that it is there. But I am never asked for any sort of project key. Am I missing something?

Clément_Stenac

It is not really possible/supported through the UI to trigger builds in other projects. I can't reproduce an issue similar to yours though

casper

Ah you are right, you confused me with jobs and projects.

I found the problem. It is indeed that you have to create build jobs inside the project they reside. You can do those in a scenario inside that project, and share the scenario to another project so it builds the shared dataset for you.

Thanks for the quick reply!