Writing dataset to Elasticsearch

Kok
Kok Registered Posts: 12

Hi,

We setup a connection in dataiku to Elasticsearch and need to write a dataset with dense_vector fields to Elastic. How can we define a dense_vector field in the dataset?

Thanks!

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,226 Dataiker

    Hi,

    dense_vector type is not support in DSS as type. I've tested to manually define the mapping on ES dataset, but it also fails Mapping update failed (HTTP code 400) {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [dense_vector] declared on field [new_column]"}].

    As a workaround you can write it as a string initially and then via the ES API, create a new index and then use ES reindex API to copy the data.

  • Kok
    Kok Registered Posts: 12

    Hi Alex,

    Thank you for your reply. In that case, I believe I have to use ES API to do the import.

  • Kok
    Kok Registered Posts: 12

    Hi,

    I am using Elasticsearch python api and try to import data from dataiku python recipe to Elasticsearch. But I kept getting "AuthenticationException(401, 'security_exception', 'missing authentication credentials for REST request " error.

    I use the same python code running on my local machine and it working fine. But the same code running in dataiku kept getting the error.

    Is there any sample code for importing data to Elasticsearch with elastic python api?

    Thanks!

Setup Info
    Tags
      Help me…