UIF + K8S + Spark fail cryptically

tihobbes

OK so without UIF, no issue with K8S+Spark. I'm using just the library from the DSS download site.

When I try to do UIF : non spark recipes run fine, but the Spark recipe no longer succeeds. The issue summary is simply :

Oops: an unexpected error occurred

The Spark process failed (exit code: 1). More info might be available in the logs.

In the log, I can see a null pointerexception, though the execution seems to keep going (and without access to the code and a debugger, I can't say much about the NPE). Much later in the log I will see an issue accessing the DKU install and data folders (DKUINSTALLDIR and DIP_HOME) then it keeps running until eventually the ProcessDied exception (com.dataiku.dip.exceptions.ProcessDiedException: The Spark process failed (exit code: 1). More info might be available in the logs.).

I'm using the recommended naming convention for automatic creation of namespaces based on user name (+ the option to make sure it's K8S compliant) as well as the auto-creation of service account.

On a side note, every time I've tried this, it seems to have done something to the user account on the Linux VM (I can't switch to it anymore etc. I've tried recreating new users that work fine until I try to run the impersonated job, then it seems to bust my user account…). That said all accounts have access to the DKU folders mentioned. Anyone has seen this before ?

Dataiku 13.4.0

Operating system used: Ubuntu

tihobbes

OK I figured it out, for my test user, its group had read ACLs on /, instead of r-x so fixing that resolved the dataiku issue.