Code Environment for Dataiku

sj0071992
sj0071992 Partner, Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2022, Neuron 2023 Posts: 131 Neuron

Hi Team,

I just want to confirm, while creating Scenarios "Sequence of Steps" I am not able to see the option of selecting Code Environment. Could you please help me with how I can select Code Environment?

Actually, I want to enable K8S for Workflow build with Visual Recipes.

Thanks in Advance

Answers

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker

    Hi,

    You can only choose python code env for the steps that require python execution. For example, custom Python step:

    Screenshot 2021-07-27 at 17.53.22.png

    or custom Python trigger:

    Screenshot 2021-07-27 at 17.56.58.png

    For build/train step, DSS will use the code env/cluster that was set within the recipe/model.

  • sj0071992
    sj0071992 Partner, Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2022, Neuron 2023 Posts: 131 Neuron

    So Does that mean like i can't use EKS for Visual Recipes?

  • Sergey
    Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker

    Hi,

    Visual recipes don't have code env selection as well as container configuration. You can only choose an execution engine (DSS/Spark/In-database, etc). Maybe that's about Code (Python/R) recipes? If we are talking about building a dataset with a Code recipe, this recipe has Advanced settings where you can set the code env and K8S cluster settings:

    Screenshot 2021-07-27 at 18.49.46.png

    As I mentioned previously, the Scenario just triggers job/activity in the DSS. When you create a step in the Scenario to run this recipe, it will be executed with the code env and within the EKS cluster you selected.

    If you have additional questions, please clarify in more detail what are trying to achieve.

  • sj0071992
    sj0071992 Partner, Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2022, Neuron 2023 Posts: 131 Neuron

    Hi,

    Actually, I have ~38 Million records in my Snowflake table to process that data in Dataiku using Visual Recipes what will be the best option in terms of Scalability and fast processing.

    Thank

Setup Info
    Tags
      Help me…