Available engines

Highlighted
tomas
Level 4
Available engines
Jump to solution
Hi,

what defines the list of available engines for data processing recipes such as prepare? I have a HDFS dataset created by Impala, then a prepare or sync to another HDFS dataset, but only Spark/MR (or LocalStream) is available. Why the DSS is not allowing to use SQL based engines? The source dataset has a hive synced table definition



Thanks
0 Kudos
1 Solution

Accepted Solutions
Mattsco Dataiker
Dataiker
Re: Available engines
Jump to solution
Hi,

For prepare recipe you only have the choice between Streaming, Hadoop/Mapreduce or Spark.

You don't have SQL engine because we generate java code we can not push in a SQL db.

https://doc.dataiku.com/dss/latest/preparation/engines.html

For sync recipe, as the purpose is to move data from one system to another, in some case we have to stream the data.

Matt

View solution in original post

0 Kudos
1 Reply
Mattsco Dataiker
Dataiker
Re: Available engines
Jump to solution
Hi,

For prepare recipe you only have the choice between Streaming, Hadoop/Mapreduce or Spark.

You don't have SQL engine because we generate java code we can not push in a SQL db.

https://doc.dataiku.com/dss/latest/preparation/engines.html

For sync recipe, as the purpose is to move data from one system to another, in some case we have to stream the data.

Matt

View solution in original post

0 Kudos