Hive metastore synchronization fails (GSS initiate failed: Server not found in Kerberos database)

UserBird

DSS 4.0

When trying to synchronize the metastore, I get this error:

[18:20:40] [ERROR] [org.apache.thrift.transport.TSaslTransport] running compute_sfpd_incidents_sample_prepared_NP - SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]
  at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
  at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
  at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
   at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
    at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
   at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
 at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:193)
 at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:155)
  at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
   at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$1.makeObject(HiveServer2ConnectionPoolService.java:171)
  at com.dataiku.dip.hive.HiveServer2ConnectionPoolService$1.makeObject(HiveServer2ConnectionPoolService.java:122)
  at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
 at com.dataiku.dip.hive.HiveServer2ConnectionPoolService.take(HiveServer2ConnectionPoolService.java:293)
  at com.dataiku.dip.hive.HiveServer2SchemaHandler.take(HiveServer2SchemaHandler.java:27)
   at com.dataiku.dip.hive.HiveServer2SchemaHandler.takeForMetastore(HiveServer2SchemaHandler.java:31)
   at com.dataiku.dip.hive.HiveServer2SchemaHandler.listHiveDatabase(HiveServer2SchemaHandler.java:40)
   at com.dataiku.dip.hive.HiveServer2SchemaHandler.isHiveDatabase(HiveServer2SchemaHandler.java:72)
 at com.dataiku.dip.hive.HiveMetastoreSynchronizer.isTableWriteSafe(HiveMetastoreSynchronizer.java:324)
    at com.dataiku.dip.hive.HiveMetastoreSynchronizer.synchronizeOneDatasetPartition(HiveMetastoreSynchronizer.java:254)
  at com.dataiku.dip.dataflow.jobrunner.ActivityRunner.waitForEnd(ActivityRunner.java:180)
  at com.dataiku.dip.dataflow.jobrunner.ActivityRunner.runActivity(ActivityRunner.java:549)
 at com.dataiku.dip.dataflow.jobrunner.JobRunner.runActivity(JobRunner.java:123)
   at com.dataiku.dip.dataflow.jobrunner.JobRunner.access$900(JobRunner.java:35)
 at com.dataiku.dip.dataflow.jobrunner.JobRunner$ActivityExecutorThread.run(JobRunner.java:312)
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
   at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:710)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
   at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
   at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
  ... 27 more
Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER
   at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:73)
    at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:191)
   at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:202)
    at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:292)
  at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:101)
   at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:456)
    at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:641)
    ... 30 more
Caused by: KrbException: Identifier doesn't match expected value (906)
   at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
    at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
 at sun.security.krb5.internal.TGSRep.(TGSRep.java:60)
 at sun.security.krb5.KrbTgsRep.(KrbTgsRep.java:55)
    ... 36 more

Clément_Stenac

The most probable cause is that the Hiveserver2 principal which you had to enter in Administration > Settings > Hadoop is not correct.

It should generally be in the form hive/fully.qualified.host.name@YOUR.REALM.COM

Note that you need the full syntax ("hive" or "hive/fully.qualified.host.name" is not enough) and that the hostname can generally not be an IP or 127.0.0.1