Data Recipe execution time is very high

[deep_215]

Hi

I have created a dataset using SQL dataset (snowflake), this dataset has 75 lakh data. In order to store this dataset in schema i am using Visual Recipe(Data preparation tool), this step is taking 1.5 hrs of time. Can anyone help with the ways to reduce the execution time.

Note:- I have tried SQL recipe but it is not working as input table in this case is not SQL table. Also i tried with Sync recipe but it is also taking 1.5hrs of time.

Regards,

Deepak

[Clément_Stenac]

Hi,

We highly recommend that you switch to a "table" dataset as input. Please note that "table" datasets can also take views, so you can define a view in your database that mirrors the query that you are currently using.

Once you have a table dataset as input, you will be able to run SQL query recipes, and you will also be able to run the Prepare recipe with "SQL" engine, which should be much faster.

Alternatively, if it is impossible to switch to "table" dataset as input, you should enable "auto fast write" on your Snowflake connection. This will allow the write-back to Snowflake to be much faster. This can be found in the settings of your Snowflake connection. You will need to have a S3 or Azure Blob or Google Cloud Storage connection in which temporary data for fast write will be written.