Announcing the winners & finalists of the Dataiku Frontrunner Awards 2021! Read their inspiring stories

Multiple Snowflake Warehouse in Single Connection

Solved!
sj0071992
Level 2
Level 2
Multiple Snowflake Warehouse in Single Connection

Hi Team,

 

I am using Snowflake Data and it is using 2x cluster configuration as my data is huge but after some filter or grouping the data is not that much huge and it will run on smaller configuration as well, so is there any possibility in recipes that we can select the warehouse so that it helps me in cost reduction?

 

Thanks in Advance

0 Kudos
1 Solution
AlexT
Dataiker
Dataiker

Hi,

Yes, your understanding is correct. If you want to run specific recipes to a different snowflake warehouse you can use the recipe variables to override the project or global variables for the warehouse value. 

Screenshot 2021-10-01 at 14.07.06.png

However, updating variables within a recursive recipe run is not really advised and is likely not needed if you can define a single warehouse for each recipe that needs to run on a different warehouse that is different from the project/global default. If you have no other choice then updating project variables during a recursive recipe run see :  https://community.dataiku.com/t5/General-Discussion/Asynchronous-project-variables-recursive-build/m...

View solution in original post

0 Kudos
4 Replies
AlexT
Dataiker
Dataiker

Hi,

You can parameterize the Snowflake connection and use variables as the warehouse value.

https://doc.dataiku.com/dss/latest/connecting/sql/snowflake.html#how-to-set-it-up

If you change the project variables or even override at the recipe level. This should allow you to achieve what you are looking for that for some recipes to use a different warehouse. 

 

 

 

0 Kudos
sj0071992
Level 2
Level 2
Author

Hi,

Just correct me if my understanding is not correct.

1. I created Connection with Warehouse as Variable

2. I run the Recipe with Large Warehouse.

3. I Update the variable to small one and then i will run my other recipes with that configuration

If my understanding is correct, then who can we automate this when we are running scenarios or running my complete Workflow at once?

Thanks in Advance

0 Kudos
AlexT
Dataiker
Dataiker

Hi,

Yes, your understanding is correct. If you want to run specific recipes to a different snowflake warehouse you can use the recipe variables to override the project or global variables for the warehouse value. 

Screenshot 2021-10-01 at 14.07.06.png

However, updating variables within a recursive recipe run is not really advised and is likely not needed if you can define a single warehouse for each recipe that needs to run on a different warehouse that is different from the project/global default. If you have no other choice then updating project variables during a recursive recipe run see :  https://community.dataiku.com/t5/General-Discussion/Asynchronous-project-variables-recursive-build/m...

View solution in original post

0 Kudos
sj0071992
Level 2
Level 2
Author

Hi, 

 

This makes sense. Thanks for the  Valuable information 🙂

 

0 Kudos
A banner prompting to get Dataiku DSS