PyCharm API not supporting PySpark recipes
Hi, the API you recently created and detailed here: https://academy.dataiku.com/latest/tutorial/code/pycharm.html does not appear to support any recipe other than made in Python.
If I connect per the instructions, I can only see recipes that were made in Python. Since there is no real difference between PySpark and Python (you can pip install the former), why do you not support PySpark? Could you not make the API ambiguous for all recipes, so therefore potentially allowing something like Scala via intelliJ?
Answers
-
I also raised the same issue with them - glad to hear I'm not alone!
-
Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 NeuronOptionsApparently not everybody agrees, but in my opinion I think @Clément_Stenac
provided a very good explanation / reason as an answer to another thread. It makes sense to be able to modify the code on one's personal favorite IDE, but then running the recipe on a local machine different than the DSS node, why would you need that? -
Are there any plans to support this? I would really like to be able to edit the recipes via the Dataiku plugin in pycharm, event if they are not immediately runnable.
