While accessing a data from a datalake , I get this error "Oops: an unexpected error occurred". HTTP Code 504, type Unknown. Not sure if there is problem at the data lake end or in dataiku. And Surprisingly it works sometimes and sometimes does not.
Without seeing the logs, it is difficult to diagnose with 100% accuracy, however the issue is quite likely caused by reverse proxy or WAF that sits between the DSS instance and the web browser. For example, the HTTP 504 error can indicate a timeout error due to queries that are too long and exceed the timeout limit. You can check for timeout errors by opening up your developer console > network tab and reproduce the error. You can record the output and send to your networking team, who would need to take a look at the relevant security rules to allow for the call.
You can also receive further guidance by opening up a support ticket with Dataiku and provide the relevant job & network logs.