Available engines

[tomas]

Hi,

what defines the list of available engines for data processing recipes such as prepare? I have a HDFS dataset created by Impala, then a prepare or sync to another HDFS dataset, but only Spark/MR (or LocalStream) is available. Why the DSS is not allowing to use SQL based engines? The source dataset has a hive synced table definition



Thanks

[Mattsco · Accepted solution]

Hi,

For prepare recipe you only have the choice between Streaming, Hadoop/Mapreduce or Spark.

You don't have SQL engine because we generate java code we can not push in a SQL db.

https://doc.dataiku.com/dss/latest/preparation/engines.html

For sync recipe, as the purpose is to move data from one system to another, in some case we have to stream the data.

Matt