Code Environment for Dataiku

sj0071992

Hi Team,

I just want to confirm, while creating Scenarios "Sequence of Steps" I am not able to see the option of selecting Code Environment. Could you please help me with how I can select Code Environment?

Actually, I want to enable K8S for Workflow build with Visual Recipes.

Thanks in Advance

Sergey

Hi,

You can only choose python code env for the steps that require python execution. For example, custom Python step:

or custom Python trigger:

 

For build/train step, DSS will use the code env/cluster that was set within the recipe/model.

sj0071992

So Does that mean like i can't use EKS for Visual Recipes?

Sergey

Hi,

Visual recipes don't have code env selection as well as container configuration. You can only choose an execution engine (DSS/Spark/In-database, etc). Maybe that's about Code (Python/R) recipes? If we are talking about building a dataset with a Code recipe, this recipe has Advanced settings where you can set the code env and K8S cluster settings:

As I mentioned previously, the Scenario just triggers job/activity in the DSS. When you create a step in the Scenario to run this recipe, it will be executed with the code env and within the EKS cluster you selected.

If you have additional questions, please clarify in more detail what are trying to achieve.

sj0071992

Hi,

Actually, I have ~38 Million records in my Snowflake table to process that data in Dataiku using Visual Recipes what will be the best option in terms of Scalability and fast processing.

Thank