Programmatic creation of Snowflake - SQl type dataset

yjagger
Level 2
Programmatic creation of Snowflake - SQl type dataset

hi,

I created a dataset using below Dataiku's Python API.

project.create_sql_table_dataset(dataset_nametypeconnectiontableschema)

 

I provide type = "Snowflake" and give the table and schema details, which already exists.

The problem is that, this operation also drops the table in Snowflake and re-creates it.

I saw in the logs, and found out that this happens when there is a schema mismatch between dataset and table.  This is from the logs 

Table already exists but with an incompatible schema: Type mismatch for column 11 (WAFER_NUM) : 'VARCHAR' in dataset, 'BIGINT'(-5:NUMBER) in table

 

Can you suggest any operations using which it does not drops the table in Snowflake, and a way for dataset to be created by automatically inferring whatever the schema is in Snowflake ?


Operating system used: Windows


Operating system used: Windows

0 Kudos
1 Reply
ZachM
Dataiker

Hi @yjagger,

Could you please provide the following information?

  1. When exactly is the table being dropped? Is it dropped immediately when you run "create_sql_table_dataset()", or does it happen at a later point?
  2. Please provide the full code that you're using to create the dataset.
  3. Where are you creating the dataset? For example, are you running the code in a notebook, or in a Python code recipe?
0 Kudos