Documentation for S3 and Snowflake Connections Setup

Solved!
NN
Documentation for S3 and Snowflake Connections Setup

Hi Dataiku Team,

We have a requirement to setup a Snowflake Connection and  a S3 connection so that we can leverage the S3 to Snowflake SYNC engine functionality.

Can you guide me to some documentation which can show me how the S3 buckets permissions and snowflake permissions etc are to be setup specifically to make the bulkcopy functionality work in the SYNC recipe.

Thanks..

0 Kudos
1 Solution
JordanB
Dataiker

Hi @NN,

Once you have configured your Amazon S3 and Snowflake connections are in DSS. Then, in order to enable fast-write, please make the following adjustments in the settings of the Snowflake connection:

  • Enable โ€œAutomatic fast-writeโ€

  • In โ€œAuto fast write connectionโ€, enter the name of the cloud storage connection to use

  • In โ€œPath in connectionโ€, enter a relative path to the root of the cloud storage connection, such as โ€œsnowflake-tmpโ€. This is a temporary path that will be used in order to put temporary upload files. This should not be a path containing datasets.

DSS will now automatically use the optimal cloud-to-Snowflake copy mechanism when executing a recipe that needs to load data โ€œfrom the outsideโ€ into Snowflake. If you set up a sync recipe from an S3 dataset to Snowflake, the engine will be set to "Direct S3 to Snowflake" automatically. 

Screen Shot 2022-09-23 at 1.26.14 PM.png

โ€ƒ

Note, the S3 and Snowflake connections must be in the same cloud region.

Please see our documentation for additional details: https://doc.dataiku.com/dss/latest/connecting/sql/snowflake.html#writing-data-into-snowflake

If you have any questions, please let us know.

Thanks!

Jordan

 

View solution in original post

0 Kudos
2 Replies
JordanB
Dataiker

Hi @NN,

Once you have configured your Amazon S3 and Snowflake connections are in DSS. Then, in order to enable fast-write, please make the following adjustments in the settings of the Snowflake connection:

  • Enable โ€œAutomatic fast-writeโ€

  • In โ€œAuto fast write connectionโ€, enter the name of the cloud storage connection to use

  • In โ€œPath in connectionโ€, enter a relative path to the root of the cloud storage connection, such as โ€œsnowflake-tmpโ€. This is a temporary path that will be used in order to put temporary upload files. This should not be a path containing datasets.

DSS will now automatically use the optimal cloud-to-Snowflake copy mechanism when executing a recipe that needs to load data โ€œfrom the outsideโ€ into Snowflake. If you set up a sync recipe from an S3 dataset to Snowflake, the engine will be set to "Direct S3 to Snowflake" automatically. 

Screen Shot 2022-09-23 at 1.26.14 PM.png

โ€ƒ

Note, the S3 and Snowflake connections must be in the same cloud region.

Please see our documentation for additional details: https://doc.dataiku.com/dss/latest/connecting/sql/snowflake.html#writing-data-into-snowflake

If you have any questions, please let us know.

Thanks!

Jordan

 
0 Kudos
NN
Author

Thank you Jordan This helps.