Submit your use case or success story to the 2023 edition of the Dataiku Frontrunner Awards ENTER YOUR SUBMISSION

How to create an Elasticsearch DSS using an index pattern

utilum
Level 1
How to create an Elasticsearch DSS using an index pattern

Due to the volume of our data, we regularly switch to writing to a new index, with a simple incremental: my_index_1, my_index_2, etc... And we have many of them. Elasticsearch API allows use of a pattern for querying, i.e. my_index_*.



Dataiku is happy connecting to a specific index, But when I try this on a working Dataiku connection the index pattren like this Dataiku refuses to save it...



What to do?

0 Kudos
1 Reply
Alex_Combessie
Dataiker Alumni

Hello,



You can achieve this by creating an index alias which uses the wildcard magic to catch the indices you want. This follows the official ElasticSearch doc: https://www.elastic.co/guide/en/elasticsearch/reference/5.2/indices-aliases.html




POST /_aliases
{
"actions" : [
{ "add" : { "index" : "test*", "alias" : "all_test_indices" } }
]
}


Then you will reference this index alias from the settings of the dataset within DSS.



Cheers,



Alex

0 Kudos

Labels

?
Labels (2)
A banner prompting to get Dataiku