Hive metastore synchronization fails (GSS initiate failed: Server not found in Kerberos database)

UserBird
UserBird Dataiker, Alpha Tester Posts: 535 Dataiker
edited July 16 in Using Dataiku

DSS 4.0

When trying to synchronize the metastore, I get this error:


[18:20:40] [ERROR] [org.apache.thrift.transport.TSaslTransport] running compute_sfpd_incidents_sample_prepared_NP - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:193)
at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:155)
at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$1.makeObject(HiveServer2ConnectionPoolService.java:171)
at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$1.makeObject(HiveServer2ConnectionPoolService.java:122)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at com.dataiku.dip.hive.HiveServer2ConnectionPoolService.take(HiveServer2ConnectionPoolService.java:293)
at com.dataiku.dip.hive.HiveServer2SchemaHandler.take(HiveServer2SchemaHandler.java:27)
at com.dataiku.dip.hive.HiveServer2SchemaHandler.takeForMetastore(HiveServer2SchemaHandler.java:31)
at com.dataiku.dip.hive.HiveServer2SchemaHandler.listHiveDatabase(HiveServer2SchemaHandler.java:40)
at com.dataiku.dip.hive.HiveServer2SchemaHandler.isHiveDatabase(HiveServer2SchemaHandler.java:72)
at com.dataiku.dip.hive.HiveMetastoreSynchronizer.isTableWriteSafe(HiveMetastoreSynchronizer.java:324)
at com.dataiku.dip.hive.HiveMetastoreSynchronizer.synchronizeOneDatasetPartition(HiveMetastoreSynchronizer.java:254)
at com.dataiku.dip.dataflow.jobrunner.ActivityRunner.waitForEnd(ActivityRunner.java:180)
at com.dataiku.dip.dataflow.jobrunner.ActivityRunner.runActivity(ActivityRunner.java:549)
at com.dataiku.dip.dataflow.jobrunner.JobRunner.runActivity(JobRunner.java:123)
at com.dataiku.dip.dataflow.jobrunner.JobRunner.access$900(JobRunner.java:35)
at com.dataiku.dip.dataflow.jobrunner.JobRunner$ActivityExecutorThread.run(JobRunner.java:312)
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 27 more
Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER
at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:73)
at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:191)
at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:202)
at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
... 30 more
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
at sun.security.krb5.internal.TGSRep.(TGSRep.java:60)
at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:55)
... 36 more
Tagged:

Best Answer

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker
    Answer ✓
    The most probable cause is that the Hiveserver2 principal which you had to enter in Administration > Settings > Hadoop is not correct.

    It should generally be in the form hive/fully.qualified.host.name@YOUR.REALM.COM

    Note that you need the full syntax ("hive" or "hive/fully.qualified.host.name" is not enough) and that the hostname can generally not be an IP or 127.0.0.1
Setup Info
    Tags
      Help me…