Code Environment for Dataiku

sj0071992
Code Environment for Dataiku

Hi Team,

 

I just want to confirm, while creating Scenarios "Sequence of Steps" I am not able to see the option of selecting Code Environment. Could you please help me with how I can select Code Environment?

 

Actually, I want to enable K8S for Workflow build with Visual Recipes.

 

Thanks in Advance

 

0 Kudos
4 Replies
sergeyd
Dataiker

Hi, 

You can only choose python code env for the steps that require python execution. For example, custom Python step: 

Screenshot 2021-07-27 at 17.53.22.png

or custom Python trigger: 

Screenshot 2021-07-27 at 17.56.58.png

โ€ƒ

For build/train step, DSS will use the code env/cluster that was set within the recipe/model. 

0 Kudos
sj0071992
Author

So Does that mean like i can't use EKS for Visual Recipes?

0 Kudos
sergeyd
Dataiker

Hi,

Visual recipes don't have code env selection as well as container configuration. You can only choose an execution engine (DSS/Spark/In-database, etc). Maybe that's about Code (Python/R) recipes? If we are talking about building a dataset with a Code recipe, this recipe has Advanced settings where you can set the code env and K8S cluster settings: 

Screenshot 2021-07-27 at 18.49.46.png

As I mentioned previously, the Scenario just triggers job/activity in the DSS. When you create a step in the Scenario to run this recipe, it will be executed with the code env and within the EKS cluster you selected.

If you have additional questions, please clarify in more detail what are trying to achieve. 

0 Kudos
sj0071992
Author

Hi,

 

Actually, I have ~38 Million records in my Snowflake table to process that data in Dataiku using Visual Recipes what will be the best option in terms of Scalability and fast processing.

 

Thank

0 Kudos