Available engines

Solved!
tomas
Level 5
Available engines
Hi,

what defines the list of available engines for data processing recipes such as prepare? I have a HDFS dataset created by Impala, then a prepare or sync to another HDFS dataset, but only Spark/MR (or LocalStream) is available. Why the DSS is not allowing to use SQL based engines? The source dataset has a hive synced table definition



Thanks
0 Kudos
1 Solution
Mattsco
Dataiker
Hi,

For prepare recipe you only have the choice between Streaming, Hadoop/Mapreduce or Spark.

You don't have SQL engine because we generate java code we can not push in a SQL db.

https://doc.dataiku.com/dss/latest/preparation/engines.html

For sync recipe, as the purpose is to move data from one system to another, in some case we have to stream the data.

Matt
Mattsco

View solution in original post

1 Reply
Mattsco
Dataiker
Hi,

For prepare recipe you only have the choice between Streaming, Hadoop/Mapreduce or Spark.

You don't have SQL engine because we generate java code we can not push in a SQL db.

https://doc.dataiku.com/dss/latest/preparation/engines.html

For sync recipe, as the purpose is to move data from one system to another, in some case we have to stream the data.

Matt
Mattsco