[CDH Cluster] Unable to start HiveServer2 Connection

BillMurray

Hi, I am evaluating DSS, so I installed it in my server and added a 2 weeks enterprise trial license.

I am facing a Cloudera CDH 5.12 cluster, kerberized.

I am able to connect and browse HDFS, but Hive connection is not working.

This is the error in log file backlog.log

[2018/05/30-07:47:19.991] [qtp1440621772-171] [INFO] [dku.hive.hiveserver2.pool]  - Creating a new HiveServer2 connection (db=default)
[2018/05/30-07:47:20.000] [qtp1440621772-171] [INFO] [dku.hive.hiveserver2.pool]  - Connecting to HiveServer2: jdbc:hive2://cdhmaster02.my.corp:10000/default;principal=hive/cdhmaster02.my.corp@MY.CORP;user=dataiku/dataiku02.my.corp@MY.CORP
[2018/05/30-07:47:20.001] [qtp1440621772-171] [INFO] [dku.hive.hiveserver2.pool]  - Driver version 1.1
[2018/05/30-07:47:20.004] [qtp1440621772-171] [DEBUG] [dku.tracing]  - [ct: 26] Done call: /api/connections/get-hive-names time=26ms user=admin thrown=CodedException
[2018/05/30-07:47:20.005] [qtp1440621772-171] [ERROR] [dip.controllers]  - API call '/dip/api/connections/get-hive-names/' failed
com.dataiku.dip.CodedException: Could not create HiveServer2 connection
    at com.dataiku.dip.hive.HiveServer2ConnectionPoolService.take(HiveServer2ConnectionPoolService.java:363)
    at com.dataiku.dip.hive.HiveServer2SchemaHandler.take(HiveServer2SchemaHandler.java:30)
    at com.dataiku.dip.hive.HiveServer2SchemaHandler.takeForMetastore(HiveServer2SchemaHandler.java:42)
    at com.dataiku.dip.hive.HiveServer2SchemaHandler.takeForMetastore(HiveServer2SchemaHandler.java:34)
    at com.dataiku.dip.hive.HiveServer2SchemaHandler.listHiveDatabase(HiveServer2SchemaHandler.java:47)
    at com.dataiku.dip.server.controllers.ConnectionsController.listHiveNames(ConnectionsController.java:113)
    at com.dataiku.dip.server.controllers.ConnectionsController$$FastClassBySpringCGLIB$$149f32e4.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:700)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
    at com.dataiku.dip.server.controllers.AuditedCallAspect.doCall(AuditedCallAspect.java:40)
    at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
    at com.dataiku.dip.server.controllers.CallTracingAspect.doCall(CallTracingAspect.java:141)
    at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:633)
    at com.dataiku.dip.server.controllers.ConnectionsController$$EnhancerBySpringCGLIB$$a73f5553.listHiveNames(<generated>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:440)
    at com.dataiku.dip.server.MissingParameterExceptionHandler.invokeHandlerMethod(MissingParameterExceptionHandler.java:18)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:428)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
    at com.dataiku.dip.server.controllers.DKUDispatcherServlet.doDispatch(DKUDispatcherServlet.java:50)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:844)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1651)
    at com.dataiku.dip.shaker.server.ResourceFilter.doFilter(ResourceFilter.java:33)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:568)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:462)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
    at java.util.Hashtable.put(Hashtable.java:464)
    at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$1.makeObject(HiveServer2ConnectionPoolService.java:202)
    at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$1.makeObject(HiveServer2ConnectionPoolService.java:128)
    at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
    at com.dataiku.dip.hive.HiveServer2ConnectionPoolService.take(HiveServer2ConnectionPoolService.java:356)
    ... 68 more

Of course I tried to impersonate the user on the machine via kinit and open a beeline session, with success. For this, I copy-pasted the connection string in the log file, and if I add double quotes in the connection string, it works. Without it doesn't (which is correct).

For the record, this command issued from dataiku's node CLI succedes:

beeline -u "jdbc:hive2://cdhmaster02.my.corp:10000/default;principal=hive/cdhmaster02.my.corp@MY.CORP"

What do I need to check ??

Thanks

Bill

Clément_Stenac

Hi,

It looks like you have added a "Connection property" in the Hive connection settings (in Administration > Settings > Hadoop) but didn't fill anything in the key. You will need to either make sure the property is filled or remove it.

BillMurray

Wooo! You're right!

May I ask you how did you found it out from the log ?

Clément_Stenac

Having the source code of DSS helps We'll make it so that it outputs a proper error message in future releases.

BillMurray

It would be really appreciated.