Programmatic creation of Snowflake - SQl type dataset

Options
yjagger
yjagger Registered Posts: 8
edited July 16 in Using Dataiku

hi,

I created a dataset using below Dataiku's Python API.

project.create_sql_table_dataset(dataset_nametypeconnectiontableschema)

I provide type = "Snowflake" and give the table and schema details, which already exists.

The problem is that, this operation also drops the table in Snowflake and re-creates it.

I saw in the logs, and found out that this happens when there is a schema mismatch between dataset and table. This is from the logs

Table already exists but with an incompatible schema: Type mismatch for column 11 (WAFER_NUM) : 'VARCHAR' in dataset, 'BIGINT'(-5:NUMBER) in table

Can you suggest any operations using which it does not drops the table in Snowflake, and a way for dataset to be created by automatically inferring whatever the schema is in Snowflake ?


Operating system used: Windows


Operating system used: Windows

Tagged:

Answers

  • Zach
    Zach Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 153 Dataiker
    Options

    Hi @yjagger
    ,

    Could you please provide the following information?

    1. When exactly is the table being dropped? Is it dropped immediately when you run "create_sql_table_dataset()", or does it happen at a later point?
    2. Please provide the full code that you're using to create the dataset.
    3. Where are you creating the dataset? For example, are you running the code in a notebook, or in a Python code recipe?
Setup Info
    Tags
      Help me…