Available engines
Options
tomas
Registered, Neuron 2022 Posts: 120 ✭✭✭✭✭
Hi,
what defines the list of available engines for data processing recipes such as prepare? I have a HDFS dataset created by Impala, then a prepare or sync to another HDFS dataset, but only Spark/MR (or LocalStream) is available. Why the DSS is not allowing to use SQL based engines? The source dataset has a hive synced table definition
Thanks
what defines the list of available engines for data processing recipes such as prepare? I have a HDFS dataset created by Impala, then a prepare or sync to another HDFS dataset, but only Spark/MR (or LocalStream) is available. Why the DSS is not allowing to use SQL based engines? The source dataset has a hive synced table definition
Thanks
Tagged:
Best Answer
-
Hi,
For prepare recipe you only have the choice between Streaming, Hadoop/Mapreduce or Spark.
You don't have SQL engine because we generate java code we can not push in a SQL db.
https://doc.dataiku.com/dss/latest/preparation/engines.html
For sync recipe, as the purpose is to move data from one system to another, in some case we have to stream the data.
Matt
