Survey banner
The Dataiku Community is moving to a new home! We are temporary in read only mode: LEARN MORE

Writing dataset to Elasticsearch

Kok
Level 2
Writing dataset to Elasticsearch

Hi,

We setup a connection in dataiku to Elasticsearch and need to write a dataset with dense_vector fields to Elastic. How can we define a dense_vector field in the dataset?

Thanks!

0 Kudos
3 Replies
AlexT
Dataiker

Hi,

dense_vector type is not support in DSS as type. I've tested to manually define the mapping on ES dataset, but it also fails Mapping update failed  (HTTP code 400) {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [dense_vector] declared on field [new_column]"}].

As a workaround you can write it as a string initially and then via the ES API, create a new index and then use ES reindex API to copy the data. 

0 Kudos
Kok
Level 2
Author

Hi Alex,

Thank you for your reply. In that case, I believe I have to use ES API to do the import.

 

0 Kudos
Kok
Level 2
Author

Hi,

I am using Elasticsearch python api and try to import data from dataiku python recipe to Elasticsearch. But I kept getting "AuthenticationException(401, 'security_exception', 'missing authentication credentials for REST request " error.

I use the same python code running on my local machine and it working fine. But the same code running in dataiku kept getting the error.

Is there any sample code for importing data to Elasticsearch with elastic python api?

Thanks!

0 Kudos