Syncing S3 data to Snowflake and skipping malformed rows

Options
yashpuranik
yashpuranik Partner, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2022, Neuron 2023 Posts: 69 Neuron

Hi All,

Suppose I have a malformed CSV in S3.

1.png

If I try to sync to Snowflake using S3 to Snowflake engine, I get the following error.

2.png

Of course I can get around it if I select DSS as the engine, but I don't want to unnecessarily use up a lot of disk space (even if temporarily) because I am looking to move massive datasets.

How do I modify the behaviour of the S3 to Snowflake stream as suggested in the error logs?

Answers

  • Sarina
    Sarina Dataiker, Dataiku DSS Core Designer, Dataiku DSS Adv Designer Posts: 315 Dataiker
    Options

    Hi @yashpuranik
    ,

    We have an existing request in our backlog to allow for use of additional Snowflake ON_ERROR options. At the moment, your only option would be to clean the data before the sync.

    I hope that information is helpful.

    Thanks,
    Sarina

Setup Info
    Tags
      Help me…