Create a SparkSQL Recipe With API

Solved!
selchheb
Level 1
Create a SparkSQL Recipe With API

Hello,

i try to create a sparsql recipe with the coderecipe createor. i saw in the library that only

"type in ["python", "r", "sql_script", "pyspark", "sparkr", "spark_scala", "shell"]:" are accepted,
i tried sql_builder = project.new_recipe("spark_sql_script", recipe_name)
i have the error that sql_builder is a none type.
Does anyone know if it is possible, and if yes what is the keyword? thanks
0 Kudos
1 Solution
SarinaS
Dataiker

Hi @selchheb,

You can use the type spark_sql_query for spark sql recipes! 

For example:

from dataikuapi.dss.recipe import DSSRecipeCreator
recipe_creator = DSSRecipeCreator("spark_sql_query", "test_recipe_creation", client.get_default_project())

recipe_creator.with_input("Orders")
recipe_creator.with_output("output_dataset_that_exists")


As a hint if you run into this again, you can always use project.get_recipe() on an existing recipe (i.e. an existing Spark SQL recipe) and then get the recipe settings, which shows the type for the recipe. 

Thanks,
Sarina

 

View solution in original post

1 Reply
SarinaS
Dataiker

Hi @selchheb,

You can use the type spark_sql_query for spark sql recipes! 

For example:

from dataikuapi.dss.recipe import DSSRecipeCreator
recipe_creator = DSSRecipeCreator("spark_sql_query", "test_recipe_creation", client.get_default_project())

recipe_creator.with_input("Orders")
recipe_creator.with_output("output_dataset_that_exists")


As a hint if you run into this again, you can always use project.get_recipe() on an existing recipe (i.e. an existing Spark SQL recipe) and then get the recipe settings, which shows the type for the recipe. 

Thanks,
Sarina