How to create an Elasticsearch DSS using an index pattern

[utilum]

Due to the volume of our data, we regularly switch to writing to a new index, with a simple incremental: my_index_1, my_index_2, etc... And we have many of them. Elasticsearch API allows use of a pattern for querying, i.e. my_index_*.

Dataiku is happy connecting to a specific index, But when I try this on a working Dataiku connection the index pattren like this Dataiku refuses to save it...



What to do?

[Alex_Combessie]

Hello,

You can achieve this by creating an index alias which uses the wildcard magic to catch the indices you want. This follows the official ElasticSearch doc: https://www.elastic.co/guide/en/elasticsearch/reference/5.2/indices-aliases.html

POST /_aliases
{
    "actions" : [
        { "add" : { "index" : "test*", "alias" : "all_test_indices" } }
    ]
}

Then you will reference this index alias from the settings of the dataset within DSS.

Cheers,

Alex